Toward Efficient Ensemble Learning with Structure Constraints: Convergent Algorithms and Applications

Abstract

Ensemble learning methods, such as boosting, focus on producing a strong classifier based on numerous weak classifiers. In this paper, we develop a novel ensemble learning method called rescaled boosting with truncation (ReBooT) for binary classification by combining well-known rescaling and regularization ideas in boosting. Theoretically, we present some sufficient conditions for the convergence of ReBooT, derive an almost optimal numerical convergence rate, and deduce fast-learning rates in the framework of statistical learning theory. Experimentally, we conduct both toy simulations and four real-world data runs to show the power of ReBooT. Our results show that, compared with the existing boosting algorithms, ReBooT possesses better learning performance and interpretability in terms of solid theoretical guarantees, perfect structure constraints, and good prediction performance. History: Accepted by Ram Ramesh, Area Editor for Data Science & Machine Learning Funding: This work was supported by the National Natural Science Foundation of China [Grants 11971374, 61772374, and 61876133]. Supplemental Material: The online appendix is available at https://doi.org/10.1287/ijoc.2022.1224 .