Performance of predictive algorithms in estimating the risk of being a zero-dose child in India, Mali and Nigeria

Abstract

IntroductionMany children in low-income and middle-income countries fail to receive any routine vaccinations. There is little evidence on how to effectively and efficiently identify and target such ‘zero-dose’ (ZD) children.MethodsWe examined how well predictive algorithms can characterise a child’s risk of being ZD based on predictor variables that are available in routine administrative data. We applied supervised learning algorithms with three increasingly rich sets of predictors and multiple years of data from India, Mali and Nigeria. We assessed performance based on specificity, sensitivity and the F1 Score and investigated feature importance. We also examined how performance decays when the model is trained on older data. For data from India in 2015, we further compared the inclusion and exclusion errors of the algorithmic approach with a simple geographical targeting approach based on district full-immunisation coverage.ResultsCost-sensitive Ridge classification correctly classifies most ZD children as being at high risk in most country-years (high specificity). Performance did not meaningfully increase when predictors were added beyond an initial sparse set of seven variables. Region and measures of contact with the health system (antenatal care and birth in a facility) had the highest feature importance. Model performance decreased in the time between the data on which the model was trained and the data to which it was applied (test data). The exclusion error of the algorithmic approach was about 9.1% lower than the exclusion error of the geographical approach. Furthermore, the algorithmic approach was able to detect ZD children across 176 more areas as compared with the geographical rule, for the same number of children targeted.InterpretationPredictive algorithms applied to existing data can effectively identify ZD children and could be deployed at low cost to target interventions to reduce ZD prevalence and inequities in vaccination coverage.