Leveraging Feature Sensitivity and Relevance: A Hybrid Feature Selection Approach for Improved Model Performance in Supervised Classification

Abstract

Many feature selection algorithms primarily give importance to identifying relevant features and eliminating redundant features. This hybrid work determines the significant features, based on the estimated individual feature sensitivities and the degree of relevance between the feature and target outcome. The majority of works currently in existence employ mutual information (MI) to calculate the degree of information between two variables. By scaling the range of the MI to [0,1], Symmetrical Uncertainty (SU) can be viewed as the normalized MI. In this proposed work, Symmetrical Uncertainty-Relevance (SU-R) is used to measure the relevance between each feature and the target outcome. Per Feature Sensitivity Analysis (PFS) is used to measure the individual feature sensitivity with the target outcome. Features are ranked based on the sum of the ranks of features calculated individually using Symmetrical Uncertainty-Relevance (SU-R) and Per Feature Sensitivity analysis (PFS). Less significant features are iteratively eliminated starting from discarding the least ranked feature identified using the combination of SU-R and PFS Analysis.To evaluate how well our proposed method identifies important features, we assess the influence of each feature on the model's performance using metrics like F1 score and accuracy. This evaluation is conducted on two diverse public datasets from the UCI Machine Learning repository, allowing us to assess the method's robustness across different data types.This hybrid work identified the best 450 significant features out of 754 in the Parkinson’s disease dataset, and the top 150 features out of 562 in the smart phone dataset. The efficacy of the SVM classifier with the selected number of significant features with the proposed hybrid PF and SU-R technique outperforms the SVM when applied with existing feature selection methods.