Evaluating Zindagi Mehfooz- Electronic Immunization Registry and suite of digital health interventions to improve the coverage, timeliness, and completeness of immunization services: A mixed methods study in Sindh, Pakistan ABSTRACT Introduction: This paper presents findings from an evaluation of Zindagi Mehfooz, electronic immunization registry (ZM-EIR) and suite of digital health interventions, aimed at improving immunization outcomes for children 12-23 months in Sindh Province, Pakistan. Methods: The mixed-methods study included a) Pre-post outcome evaluation using vaccine coverage from the Multiple Indicator Cluster Surveys (MICS) 2014 and 2019; b) In-depth interviews with caregivers, vaccinators, supervisors, and managers; c (Preprint)

Abstract

Zindagi Mehfooz (Safe Life) Electronic Immunization Registry (ZM-EIR) in Sindh Province, Pakistan is a comprehensive suite of digital health interventions that aims to improve equitable access, timeliness, and coverage of child immunizations through a smartphone-based application (app) for vaccinators, web-based dashboards for supervisors and managers, text message alerts and reminders for caregivers, and a call center. EIRs are designed to improve the collection, analysis, and use of immunization program data to improve efficiency, equity, and coverage of immunization programs [1]. Improved access to better data can lead to better decisions, planning and operations, which in turn will lead to better health outcomes [2]. ZM-EIR is comprised of complementary digital health interventions that satisfy and go beyond the conventional definition of an electronic immunization registry (EIR). Because EIRs are intended to support immunization programs with improved data-driven decision-making, the evaluation of data use is also important [3]. Drivers of EIR use include organizational factors (supervision, support, resources, facility type, location, and volume of clients), technical factors (infrastructure, electricity and internet connectivity), and behavioral factors (training, capacity, and motivation of system users) [3]. A recent study using EIR data to systematically evaluate the timeliness of vaccination found mixed results with the conclusion that the overall reliance of the study on system-level data was insufficient to evaluate its impact on immunization outcomes [4]. In a semi-systematic review of the evidence, there are a moderate number of process evaluations, few pilot studies, and no large-scale effectiveness studies of EIRs. ZM-EIR aims to strengthen EPI service delivery and health system by addressing supply and demand-side barriers that contribute to sub-optimal utilization of immunization services. On the supply side, it is designed to overcome the inability to track immunization history for individual children and assess catch-up immunization schedules; lack of motivation among health workers; excessive burden of paperwork; and poor reporting, management, supervision, and monitoring of vaccinators. On the demand side ZM-EIR is designed to overcome challenges associated with low uptake of immunization services due to a lack of awareness and motivation among parents as well as inability to remember vaccine appointments. In 2012-13, only about half (58.3%) of children (12-23 months) received all basic vaccinations , according to Pakistan Demographic and Health Survey 2012-13, with marked geographic variation across the country; Sindh Province had a full immunization coverage of 35% in 2014, according to the Multiple Indicator Survey. Since its inception in 2012 until August 2022, ZM has scaled across 30 districts of Sindh Province with phased introduction in other areas including the Federal Capital (Islamabad), Khyber Pakhtunkhwa province and Gilgit-Baltistan region and has been used by over 5,000 vaccinators at public and private immunization clinics to enrol over 7 million children.

This paper summarizes findings from a mixed-methods study of ZM-EIR which assessed vaccinator use of ZM-EIR, provider and caregiver perceptions of ZM-EIR, temporal changes in immunization coverage and timeliness among children 12-23 months, equity, and cost-effectiveness.

Intervention description ZM-EIR’s functionality centers around a mobile phone-based registry with a suite of complementary interventions. See Figure 1 for a graphical overview of the ZM system. ZM-EIR functionality 1) Web-interface for data visualization 2) Mobile based data entry and access for vaccinators 3) Geo-spatial data for vaccination events 4) 2-way interactive SMS reminders for parents/caregivers 5) Call center to answer queries of parents/caregivers 6) Offline mode to work in areas of poor connectivity 7) Child registry for enrolling newborns or never-vaccinated children 8) Immunization decision support system (iDSS) to guide vaccinators on routine and catch-up immunizations 9) Defaulter reports for vaccinators 10) Gamified videos for training vaccinators 11) AI-based predictive analytics to identify children highly likely to drop out 12) AI-based chatbot to address caregiver queries and concerns in real-time 13) GSM-based tracking of vaccinators during work hours 14) QR code-based tracking for unique identification of children and women The mobile Android app is used to capture data for each point of contact between the immunization service, caregivers and their children. At the time of first enrolment the child’s data are entered on the paper Expanded Program for Immunization (EPI) card and entered into ZM-EIR. The paper home-based record is linked to the electronic system by a unique QR code. During follow-up visits, the child’s ID is retrieved by scanning the QR code, giving the vaccinator access to an individual child’s vaccine schedule. For each vaccination received, the data is entered into the ZM app and linked to the server. All data entered into the ZM app are collated on a central web-based dashboard which supervisors can use to monitor progress, identify defaulters, and take corrective action to achieve targets. To remind caregivers of when they need to next visit the clinic, up to three automatic personalized SMS reminders are sent one day before, on the day of, and six days after the scheduled immunization date (if the child did not make the scheduled appointment). A child is considered to have completed the full course of immunizations after the second dose of measles. Study design A pre- / post- study mixed-methods design measured changes in the proportion of 12-23 months children fully immunized from 2014 to 2019. Additional aims and objectives of the study were measured through qualitative research (observation and in-depth interviews), secondary data analysis (Multiple Indicator Cluster Surveys (MICS) from 2014 and 2019) as well as comparison between districts and health facilities with high, moderate, and low ZM-EIR compliance and high, moderate, and low immunization program performance. The results have been presented in the findings section by study aim and objective. Experimental or quasi-experimental study designs were not feasible as ZM-EIR was implemented throughout Sindh Province and a suitable comparator was not available. Patient and Public Involvement There was no patient or public involvement in the design or implementation of the study. Caregivers, health workers, and health administrators were included in the qualitative research component as key informants after providing informed consent. The study received IRB approval from the IRD Review Board with number – IRD_IRB_2021_11_001 and is registered with ISRCTN #23078223. Data Sources and Analysis The two sources of quantitative data are the Multiple Indicator Survey conducted across the state of Sindh, Pakistan by UNICEF from 2014 and 2019, and the database of the ZM registry. Details of the MICS survey are provided on the UNICEF website. The indicators across the study aims and objectives are listed in Table 1. Table 1: Indicators included in Quantitative Analyses Indicator Definition Numerator Denominator Data source Compliance Vaccinator updates/uploads 1 record per vaccination day Number of days the vaccinator logged in the ZM app and uploaded/updated at-least one child record Number of vaccination days for that health center/vaccination center ZM database ZM registry coverage The total number of children 12-23 months receiving any immunization reported in the ZM Estimated monthly target population of children of Sindh Province that is derived by the EPI program in close working/collaboration with the ZM team ZM child records database ZM registry use The proportion of vaccinators trained to use ZM who upload details for at least 1 client per day in Sindh from 2017 to 2019 Number of days vaccinator logged in the ZM app and uploaded/updated at-least one child record Number of vaccination days for that health center/vaccination center ZM vaccination staff database ZM dashboard use The number of unique user (Manager/ Supervisor) log-ins to the dashboard Number of log-ins per district per year ZM management staff usage dashboard derived from ZM database Fully immunized Child has received BCG at birth, OPV 1, 2, and 3, Penta 1, 2, and 3, Pneumococcal 1, 2, and 3, and Measles 1 in 12-23 months old children, then child will be considered as fully immunized Number of children 12-23 months who received BCG at birth, OPV 1, 2, and 3, Penta 1, 2, and 3, Pneumococcal 1, 2, and 3, and Measles 1 Number of children 12-23 months sample as part of the survey in Sindh province MICS 2014, 2019 Zero-dose Child aged 6 to 23 months that has not received the first dosage of DPT1/Penta 1 Number of children who did not received DPTI/Penta I Vaccine-wise Timely immunization Child has received the dose of antigen at the recommended age specified in the EPI schedule Number of children 12-23 months who received BCG at birth, OPV 1, 2, and 3, Penta 1, 2, and 3, Pneumococcal 1, 2, and 3, and Measles 1 as per the timeliness criteria of the Pakistan / Sindh province EPI program We use survey design adjusted estimates of vaccine coverage estimated according to guidelines provided by MICS. The relative odds ratios for the timeliness of the different antigens was assessed using regression models (logit) with the outcome variable of receiving an antigen within the appropriate timeframe. These models are adjusted for wealth, education, residence, and number of children in households and 95% confidence intervals adjusted for clustering at the community level (PSU) are presented. Data on program costs were provided by IRD and drawn from financial records maintained by the program for the analytic time horizon of October 1, 2017- February 28, 2019. Costs were categorised into major programmatic activities and into one of two cost-categories: capital costs (costs with a life expectancy of >1 year) or recurrent costs. Capital were annualized using an annualization factor [((1+r)^n ) - 1] / [r(1+r)^n], where r = discount rate of 3% and n = useful life of the capital item. All capital items, including initial training, servers, and mobile phones were assumed to have a life expectancy of five years, a window of time consistent with the observed useful life of these items in the field. Cost are presented in 2019 USD using consumer price indices and foreign exchange rate from oanda.com for 2019. Incremental changes in vaccination coverage were drawn from MICS analyses based on those antigens observed to have a statistically significant change in coverage. Changes in coverage were inputted in the Lives Saved Tool [1] and used to estimate lives saved as a result of ZM implementation. Programmatic costs were drawn from IRD financial records. Qualitative data were collected among caregivers of children <2 years (n=12), vaccinators (n=12), supervisors (n=3), district level EPI managers (n=7), provincial EPI managers (n=2), and ZM-EIR implementation and data teams (N=4). Thematic analysis was conducted using Grounded Theory through a two-step process: 1) iterative debriefs conducted concurrently with data collection and 2) through coding of transcripts and generating sub-themes and themes. A theoretical framework based on these themes and sub-themes was then developed in line with the research study objectives and used to analyze and present findings.

ZM-EIR reach and use by providers Vaccination staff training on ZM-EIR started in Quarter 3 of 2017 as part of a district-wide planning with the Sindh Provincial Department of Health. Data on the proportion of vaccination staff trained on the system out of the total were not available. However, of the 1307 staff members nominated for training, all (100%) were trained across 52 batches between September and December of 2017. Training batch sizes ranged from 12-40 participants per training. Sindh Provincial EPI defines compliance with the ZM-EIR system by vaccinators as one upload or update for at least one client per vaccination day. Compliance estimates include instances in which a vaccinator may be present at the health facility, but no child is brought in for vaccination. Findings suggest that across 28 districts, compliance rates ranged from 46% to 85%. The average compliance increased from 40% in 2018 to 57% in 2019. Data is available in Supplemental Material. Trends in compliance were assessed based on demographic characteristics of vaccinators. Compliance and usage during 2017-2019 was slightly higher among urban vaccination staff compared to their rural counterparts. Compliance among female vaccinators was higher as compared to males even though the proportion of females in the vaccination staff is only 15%. There was a gradual increase of compliance in all age-groups during 2017 to 2019. Younger age groups (less than 40 years of age) showed improvement from 40% in 2017 to 49% in 2019 with the highest proportional increase observed for those aged 26-30 years (49%) and those aged 18-25 years (47%). More modest increase was found for vaccinators in higher age groups with the 51-55 year group showing only 28% increase. Stakeholder perceptions of ZM-EIR Stakeholder perceptions of ZM-EIR were assessed for each set of functions within the ZM-EIR suite of digital health interventions, including the perceived benefits and challenges of use to improve immunization service delivery (Box 2). Overall, all stakeholder groups interviewed had a favorable impression of ZM-EIR and the associated suite of interventions. Provincial managers felt that the real-time access to vaccination data facilitated longitudinal monitoring of vaccination coverage over time and across geographic areas. They further noted that expansion of ZM-EIR would allow this to occur across the country as it would enable the provinces to see and evaluate each other’s coverage. ZM-EIR was thought to provide a blueprint of what a national repository of EPI records could look like and further be a mechanism for improving supply monitoring and fraud prevention. As the primary end-users of ZM-EIR, vaccinators expressed their satisfaction with ZM-EIR features, specifically for defaulter and zero dose lists. Key Informant General Perceptions of ZM-EIR Caregiver Perceptions ‘... it is very helpful. Otherwise, we are busy with our work and forget about it. When we receive a message then we know we have to get the child vaccinated soon.’ The caregivers who used the call center, called it, ‘good in every way’. According to them, these have been proved useful in gaining information about the availability of the vaccination teams, discussing any symptoms caused by the vaccination, and asking for a second dose. Vaccinator Supervision ...‘wherever he (vaccinator) goes, we monitor it through Google Maps When we visit a vaccinator, it is a surprise visit. We do not call them. We just follow them on mobile and visit them on field.’ ‘These graphs depict which vaccinators have complied [based on required vaccinations for the day], and which are still non-compliant.’ Manual (Paper-based) Data Capture ‘our people in the field write in the book, on the card, and all that writing is very difficult. It would be better if we just use ZM-EIR, and only one thing stays.’ “if they had owned it properly, it would end. The vaccinators who work honestly in the field ask us to put an end to the manual data capture work so they can continue working with ZM only.’ ‘all their registers will cease to be used, and all their data will be done on ZM. That will be their daily register and stock register. Work is being done on this, and I think in one or two months they may become paper-free’ Registry coverage Registry coverage was calculated using the enrolment numbers for each calendar year birth cohort as numerator and the annual targets provided by the EPI (estimated population based on modeling using birth rates and recent census figures) as the denominator. The target population of children is derived based on the enrolment of children at the time of birth through a birth cohort registration and an EPI schedule-based continuum of care mechanism. From 2017 to 2019, the ZM-EIR immunization registry enrolled nearly 5.4 million children and 1.9 million married women of reproductive age group (women aged 15-49 years). Gender-related barriers to registry uptake - Gender was identified to play a significant role in ZM-EIR uptake by caregivers owing to its embeddedness within the local, rural contexts. There is often a clear distinction between caregiver’s interaction with male vaccinators and that with female vaccinators. One key informant emphasized the need for lady health workers who can bond with female caregivers (who are usually more responsible for taking care of their children than male caregivers) to resolve any resistance or anxieties regarding vaccinations in an attempt to further expand immunization efforts. A female caregiver shared that ‘the female health worker can come inside the house, and we [caregiver] know her so it’s good that a female health worker comes. If a man comes then it is quite difficult’. These deeply embedded gendered preferences related to vaccination experiences reinforce the need to incorporate cultural and gender considerations when designing digital health interventions. ZM-EIR Data Use The provincial managers shared that the ZM-EIR app provides defaulter and zero dose lists which help them meet their target and improve performance. It enables those who are supervising to encourage the vaccinators to bridge the gaps in the coverage of immunization targets. The data is reliable because it is not dependent on the availability of internet services. This data is then communicated to other responsible representatives at the district and provincial levels. Among the stakeholders who can access the data reports and dashboards, the role of supervisors has been seen to be relatively limited compared to others. The supervisor’s overall responsibilities include monitoring and providing support to vaccinators, checking vaccination stock, and planning and conducting field visits. For their limited use of ZM-EIR app to monitor the activities of vaccinators in their respective district areas, it was stated that newer and younger supervisors are able to use the app comfortably while older supervisors have found the app difficult and less engaging. Coordinators and supervisors also suggested that the latter should be given access to a separate app to monitor the vaccinators so that they do not have to rely on the district managers for the data. Observation of data entry and use - Direct observations of immunization services were conducted for one day per facility in six health facilities and three district offices. Among the health facilities observed, half of the facilities observed were clean, organised, and relatively well-maintained. The other half were seen as unclean, ‘cluttered’, and with ‘broken furniture’. The number of rooms in a health facility ranged from two to seven. All the health facilities provide vaccination and other health services. Most of the health facilities had two vaccinators present at the facility. Of these one in each facility used ZM-EIR for registration and defaulter lists. In all six facilities, it was reported by vaccinators that when the caregivers would bring their child for vaccination, they did not use ZM-EIR for registration in real-time. Only four out of six facilities had electricity, one being backed by a solar power system. Network connectivity was limited in all facilities which also hindered the use of ZM-EIR in real-time. All district offices observed had one supervisor on-site with access to a computer / laptop in the office which enables access to web-based ZM-EIR dashboards. The supervisors reported to engage in daily monitoring of the vaccinators whereby they track the vaccinators’ attendance, location, and registration. No other use of ZM-EIR was reported or observed. Key Informant Views of ZM-EIR Data Use by Key Stakeholder Group District Field Coordinators and Managers ‘when we have their attendance in the morning, the supervisors call right then to ask why this vaccinator isn’t marked present, tell their timing and all. So, we talk to them daily’. ‘I can also see everything about defaulter children and why we can’t cover them.’ Supervisors ‘[supervisors] do monitoring by checking their vaccines, by checking their temperature in the monitoring. We check the graph and record to check whether they are taking records or not. We also check whether they [vaccinators] use ZM or not. This is our work’ ‘we can’t tell if it is useful for the supervisors. Because we don’t know where the vaccinators are. We should be able to check their work through it, even as a guest or monitors.’ ‘all supervisors are old-aged. If we make them use it, they won’t be able to. They can’t even see the dashboard, because those poor people aren’t used to it since the start’ Vaccinators ‘the vaccinators are scared because when their defaulters increase, their UC goes into categorization B or C. So they are working on the coverage of defaulters because with ZM we are getting proper data now. Since the defaulters are being covered to keep their UCs up, that means vaccinations are happening timely’. Vaccination coverage The vaccine-wise coverage across this evaluation’s time frame of consideration (2017-2019) reflects that since its inception for scale up in the province and phase-wise implementation in all districts; increases were observed for Penta III and Measles I vaccines, with the highest increments (12%) observed in Measles I coverage (the last EPI schedule-based vaccine) . The prevalence of full immunization among children 12-23 months, went up from 5% to 15.1% between 2014 and 2019 on average by district. Vaccination timeliness was assessed by antigen by comparing the proportion of children 12-23 months immunized across MICS surveys in 2014 and 2019. Findings suggest that improvements in timeliness were achieved for BCG (+8.3%), Penta I (+3.2%) and Measles II (+1.3) vaccines. Reductions were seen for Penta II/DPT II (-49%) and Measles I (-4%). DPTII / Penta II is a challenging indicator for timeliness as it has the most stringent of conditions, needing to be exactly midway between I and III. Key Informant Views of ZM-EIR Relationship to Immunization Coverage and Equity Immunization Program Performance ‘…in what [performance] category are [2] UCs, i.e., whether [they] are in red, green, blue, or yellow . . . every UC-wise, every vaccinator-wise, every town-wise, every district-wise, we are getting all the data’. Immunization Coverage ‘Because of ZM, the EPI office has found out that we have 114 vaccinators. If only 67 vaccinators out of them are working, they ask your district where the rest of your vaccinators are. They ask us to enroll them, assign them a field, and send them to work. So now I have over a hundred vaccinators enrolled, and they work themselves after returning. Coverage will thus ultimately increase’. Immunization Equity ‘in outreach, more female children are being vaccinated because they find it easy that they can get the immunization service at their doorstep . . . we can see [in defaulters list] that more females might be lagging behind as a result of this, there is catch-up, and we do see better coverage among the female population’. Immunization Timeliness ‘. . . three SMS reminders are sent so it’s not that you only get one SMS reminder for the child’s appointment day . . . there are these three different time points as a constant nudge to the caregiver to bring the child for immunization at the time that he is required to show up’. The increase in immunization coverage may also be attributed to more structural factors e.g. public trust in government. From a ground level perspective, vaccinators shared that the selection of local people for authority positions in rural governments has increased the trust of people in government, leading local caregivers to trust health authorities and their immunization efforts. In general, the vaccinators also believe that there has been an increase in awareness coupled with an increase in diseases which has had a concurrent effect on improved vaccination rates among rural communities. Managers shared that timeliness of immunization may be due to the decision-support system built within the app. While the SMS reminders as a ZM-EIR feature are more caregiver-oriented, the decision-support system within the app integrated as an explicit function is more vaccinator-oriented. In the instances where vaccinators haven’t had substantive training, the vaccinator is likely to get confused about which vaccine to give the child at a specific time. The decision-support system uses a built-in algorithm to automatically calculate which vaccine is to be given based on the date of birth of the child and the immunization history. This minimizes the chance of missed immunization and enables the vaccinators to provide age-appropriate and timely immunization. Zero-dose and timeliness The prevalence of zero dose children 6-23 months, defined as not receiving at least one dose of Pentavalent, went down from 38.7% to 24.7% between 2014 and 2019. The changes were similar across the board in terms of sociodemographic characteristics. Timeliness increased by 1.5-2 times for BCG, Penta1, Penta2, Measles1 and Measles2, while the models show a decrease for PCV3 and Penta3. 1. PCV was not available widely in 2014 and hence the dramatic increase in 2019. Program Costs, Cost per life saved Overall program costs from October 1, 2017 to February 28, 2019 were $3.27 million USD. The Start-up phase – characterized by one-time activities required to initiate the program occurred from October 2017 to February 28, 2018 and cost an estimated $2.21 million USD. Nearly two-thirds of start-up costs were comprised of equipment and personnel. The former, included costs associated with two servers, 47 laptops, and 3,837 phones and accessories; items procured by donors directly and donated to the project. By the end of February 2018, ZM-EIR had been rolled out across 27 districts in Sindh. Implementation costs were assessed for a 12-month window from March 2018 to February 2019. The leading cost drivers were personnel including technical and field staff (53%), refresher training and support travel (11%), equipment (10%), airtime (8%), and initial training costs (6%). Initial training spanned for 2-3 days per town and 3-6 towns per district, and included travel, personnel costs, and per diems for trainees. Airtime costs included mobile airtime used to support program monitoring and implementation. Replacement costs were estimated to account for equipment provided by donors and noted above. A 10% indirect cost was incorporated across all reported costs. Table 2 summarizes incremental lives saved for 2017-2019 following the rollout of ZM across Sindh. Despite the slight decline in immunization coverage for pentavalent, the population increase in the number of children immunized even at lower rates, still corresponded to an incremental increase in lives saved for Hib-3 and DTP-3. The latter comprised 41% and 12% of total lives saved in 2018, respectively. Table 2 Incremental lives saved from 2017-2019 among children 12-23 months in Sindh Antigen 2017 2018 2019 TOTAL BCG - Single dose 0 0 0 0 Polio - Three doses 0 0 0 0 DTP - Three doses 96 132 138 366 Hib-Three doses 386 463 482 1330 HepB - Three doses 0 0 0 0 Pneumococcal - Three doses 251 366 410 1027 Measles - Single dose 118 172 176 466 TOTAL 851 1 133 1 205 3 189 Program costs include annualized Start-up costs and Implementation costs for the March 1, 2018 to February 28, 2019. Results suggests that the total annualized program cost for 12-months in 2019 USD was $1.54 million USD corresponding to a cost-per life saved of $1,362 USD.

This external evaluation of the ZM-EIR and associated suite of digital health interventions and for childhood immunization in Sindh, Pakistan highlights important outcomes, insights and recommendations across stakeholder groups. It is worth noting that the decrease in uptake of ZM-EIR between 2018-2019 in certain districts may reflect a catchup early in implementation of the backlog of children from previous years who would otherwise not have been vaccinated but are getting vaccinated at a later age with the implementation of ZM-EIR. Study findings suggest that the prevalence of fully-immunized children 12 to 23 month increased by 10%, from 5% before implementation to 15.1%immunized after 3 years of implementation. The rate of on-time vaccination (timeliness) improved for most but not all antigens in the recommended childhood vaccination schedule. The prevalence of zero-dose children between 6 and 23 months of age decreases from 38.7% pre-implementation to 24.7% after implementation. Increases in immunisation coverage are estimated to result in over 3,000 lives saved; corresponding to a program cost per life saved of $1,362 USD. While changes in coverage and immunization timeliness cannot be attributed wholly to the ZM-EIR program, study findings are a promising indication that EIRs can support improvements in child health outcomes. Qualitative findings support this assertion; indicating that, overall, stakeholders interviewed had a favourable impression of the ZM-EIR suite of interventions. Several suggestions and areas for improvement were made by key informants to address some of the challenges experienced by users of ZM-EIR. Firstly, gender dynamics, gender segregation (male vaccinators / female caregivers), education, literacy, and cultural norms impacted access and use by vaccinators and the ability to make direct contact with caregivers. Although their numbers are lower, female vaccinators have higher compliance with the ZM-EIR system than their male counterparts. Power, trust, and relationships were noted to present challenges between vaccinators and clients of the opposite gender. The implementation of ZM-EIR brings to the fore the deep-rooted gender inequities in society that need to be addressed through relevant policy measures to leverage the full potential of the intervention. In addition, gender analyses and planning at the start of future ZM-EIR intervention planning may also help identify strategies to help overcome this gap. Secondly, we noted reliance on double record-keeping which needs to end to reduce the data-entry burden (double requirement to complete both paper and mobile data entry) for field vaccinators and facility staff and recommend moving towards a paperless system. Our findings show that the use of ZM-EIR cannot be fully maximized unless physical record-keeping is phased out as it impacts the potential impact the ZM-EIR app can have on immunization service delivery. Small pilot studies in countries like Viet Nam and Bangladesh with immunization programs rolling-out EIRs report improvements in key immunization program milestones such as timeliness of immunization (vaccinations delivered on-time according to established schedule recommendations), a reduction in vaccination schedule drop-outs and an improved ability for vaccinators to identify and follow-up with defaulters [5-8]. Similar to this evaluation, health workers using EIR systems report that they use the data to identify defaulters, coverage disparities and vaccine stock levels; and that they have more confidence to take action based on these data analyses [6]. However, this is the first known study of this degree of comprehensiveness, scale, and rigor to evaluate an EIR. In addition to the inclusion of standard EIR features (as illustrated in Table 8) [6], ZM-EIR includes AI-based predictive analytics to identify children more likely to drop out; automated SMS reminders, call center, and AI chatbot for caregiver questions, concerns, and complaints. The suite of interventions also includes geo-spatial data for vaccination events - geospatial data to track each child’s immunization progress, real-time tracking of vaccinators with GIS tracking to improve monitoring and accountability, simultaneous supervision of multiple teams, monitor attendance and mobility of field staff, as well as gamified videos and training for vaccinators. Table 3 Comparison of ZM and ideal EIR characteristics and functionalities Features of Zindagi Mehfooz -EIR (ZM-EIR) Ideal requirements of EIR [9] ZM-EIR is a registry to enroll all newborns or never-vaccinated children with a unique ID / QR code assigned to each woman and child; a web-interface allows data visualization, mobile-based data entry and access for vaccinators Registration and search features: Enrollment at birth, Client management, Unique and unequivocal ID Registration includes all relevant demographic and location data of patient and facility with geo-location; barcoded vaccine vials linked with vaccine record; Offline mode for areas with low mobile/data coverage [not linked to other health areas] Patient records: Individual demographic data, Vaccine event data, Nonroutine (campaign and outreach) vaccine events, and linkages to other health areas Immunization decision support system guides vaccinators on routine and catch-up immunizations; Defaulter reports for vaccinators; Interactive SMS reminders for parents/caregivers Vaccination monitoring and follow-up: Clinical decision support, Identification of under-vaccinated children/missed vaccination, Reminder and recall messages Vaccinators are registered within the system and assigned to a health facility registered to a district management unit. Health facility registration and management Vaccine stock is tracked through number and type of vaccinations given by vaccinator, facility, and district. Stock management Geospatial analysis to map pockets of under-vaccinated children with exact location to enable follow-up with refusal households; Call center for complaints and AEFI reporting Data and reporting: Data aggregation at different geographic and administrative levels, Adverse event reporting Follows industry standards for data security and privacy platform interoperable with DHIS2 and Government’s EPI-MIS deployed at full scale in 29 districts of Sindh Province by over 3,096 vaccinators at 1,694 public and private sector immunization clinics to enroll 3.8 million children and 1.3 million women with 31 million immunization visits recorded (as of 2021). Other system requirements: Data exchange and interoperability, Offline capability, Alignment with international standards, Data privacy and security, Scalability and capacity, Usability Limitations of the study Intervention cost-effectiveness relies on the assumption that changes in immunization coverage were attributed to ZM-EIR and no other factors including health systems strengthening, or other immunization initiatives. In addition to this limitation, efforts to determine the incremental cost-effectiveness are hampered by the absence of a true ‘comparator’ or control area or population with data on both costs and consequences. CONCLUSION This study contributes to the evidence base of the added value and recommendations for implementing EIRs in LMICs. Study methods included a retrospective analysis of pre-existing survey and system generated data, as well as qualitative stakeholder interviews. Findings suggest that ZM-EIR is a promising technology platform which may have contributed to improvements in the timeliness and coverage of childhood immunisations as well as reduction of zero-dose children in Sindh Province, Pakistan from 2017-2019.

The study received IRB approval from the IRD Review Board with number – IRD_IRB_2021_11_001 and is registered with ISRCTN #23078223.