A Model-Free Feature Selection Technique of Feature Screening and Random Forest-Based Recursive Feature Elimination

Abstract

This paper studies data with mass features, commonly observed in applications such as text classification and medical diagnosis. We allow data to have several structures without requiring a specific model and propose an efficient model-free feature selection procedure. The proposed method can work with various types of datasets. We demonstrate that this method has several desirable properties, including high accuracy, model-free, and computational efficiency and can be applied to practical problems with different modelings. We prove that the proposed method achieves selection consistency and $L_2$ consistency under mild regularity conditions. We conduct simulations on various datasets, including data generated from the generalized linear model, additive model, Poisson regression, and binary classification model. These simulations illustrate the superior performance of the proposed method compared to other existing methods across different model settings. In addition, we apply our method to two real examples, the Tecator dataset and the Daily Demand Orders dataset, both of which are continuous and high dimensional. In both cases, our method consistently achieves high accuracy in prediction and model selection.