Evaluating the Performance of Common Machine Learning Classifiers using various Validation Methods

Abstract

The selection of the proper classifier and the implementation of the proper training strategy have the most impact on the performance of machine learning classifiers. The amount and distribution of data used for training and validation is another crucial aspect of classifier performance. The goal of this study was to identify the optimal combination of classifiers and validation strategies for achieving the highest accuracy rate while testing models with a small dataset. To that end, five primary classifiers were examined with varying proportions of training data and validation procedures. Most of the time, Random Forest and Nave Bayes classifier models outperformed competing classifiers. However, we discovered the best performance when we employed the holdout cross-validation technique using 70% of the available data as a training set and the remaining data as a test set.