Fuzzy Evaluation and Benchmarking Framework for Robust Machine Learning Model in Real-Time Autism Triage Applications

Abstract

AbstractIn the context of autism spectrum disorder (ASD) triage, the robustness of machine learning (ML) models is a paramount concern. Ensuring the robustness of ML models faces issues such as model selection, criterion importance, trade-offs, and conflicts in the evaluation and benchmarking of ML models. Furthermore, the development of ML models must contend with two real-time scenarios: normal tests and adversarial attack cases. This study addresses this challenge by integrating three key phases that bridge the domains of machine learning and fuzzy multicriteria decision-making (MCDM). First, the utilized dataset comprises authentic information, encompassing 19 medical and sociodemographic features from 1296 autistic patients who received autism diagnoses via the intelligent triage method. These patients were categorized into one of three triage labels: urgent, moderate, or minor. We employ principal component analysis (PCA) and two algorithms to fuse a large number of dataset features. Second, this fused dataset forms the basis for rigorously testing eight ML models, considering normal and adversarial attack scenarios, and evaluating classifier performance using nine metrics. The third phase developed a robust decision-making framework that encompasses the creation of a decision matrix (DM) and the development of the 2-tuple linguistic Fermatean fuzzy decision by opinion score method (2TLFFDOSM) for benchmarking multiple-ML models from normal and adversarial perspectives, accomplished through individual and external group aggregation of ranks. Our findings highlight the effectiveness of PCA algorithms, yielding 12 principal components with acceptable variance. In the external ranking, logistic regression (LR) emerged as the top-performing ML model in terms of the 2TLFFDOSM score (1.3370). A comparative analysis with five benchmark studies demonstrated the superior performance of our framework across all six checklist comparison points.