Predicting outcomes of smoking cessation interventions in novel scenarios using ontology-informed, interpretable machine learning

Abstract

Background Systematic reviews of effectiveness estimate the relative average effects of interventions and comparators in a set of existing studies e.g., using rate ratios. However, policymakers, planners and practitioners require predictions about outcomes in novel scenarios where aspects of the interventions, populations or settings may differ. This study aimed to develop and evaluate an ontology-informed, interpretable machine learning algorithm to predict smoking cessation outcomes using detailed information about interventions, their contexts and evaluation study methods. This is the second of two linked papers on the use of machine learning in the Human Behaviour-Change Project. Methods The study used a corpus of 405 reports of randomised trials of smoking cessation interventions from the Cochrane Library database. These were annotated using the Behaviour Change Intervention Ontology to classify, for each of 971 study arms, 82 features representing details of intervention content and delivery, population, setting, outcome, and study methodology. The annotated data was used to train a novel machine learning algorithm based on a set of interpretable rules organised according to the ontology. The algorithm was evaluated for predictive accuracy by performance in five-fold 80:20 cross-validation, and compared with other approaches. Results The machine learning algorithm produced a mean absolute error in prediction percentage cessation rates of 9.15% in cross-validation, outperforming other approaches including an uninterpretable ‘black-box’ deep neural network (9.42%), a linear regression model (10.55%) and a decision tree-based approach (9.53%). The rules generated by the algorithm were synthesised into a consensus rule set to create a publicly available predictive tool to provide outcome predictions and explanations in the form of rules expressed in terms of predictive features and their combinations. Conclusions An ontologically-informed, interpretable machine learning algorithm, using information about intervention scenarios from reports of smoking cessation trials, can predict outcomes in new smoking cessation intervention scenarios with moderate accuracy.