Reflections on Required Competencies For Health Systems Monitoring, Evaluation, and Learning In India

Abstract

The movement towards systems thinking and complexity-informed monitoring and evaluation has been ongoing for some time. Covid-19 has accelerated this shift and increased the salience of contextually aware, adaptive forms of evaluation. Drawing from our experience over three years of providing monitoring, evaluation, research and learning (MERL) to a large international philanthropic organization’s health systems portfolio of work in India, we identify three types of evaluator capacities and associated set of competencies that derive from complexity-driven practice in a developing country. The first type of capacity relates to deriving evidence in contexts where there are very few traditional forms of data. The MERL portfolio related to health systems outcomes of access, financial protection, and equity even as the health programs themselves were evolving in a period of rapid transformation and dynamism of policy priorities and implementation mechanisms. This required an integration of traditional performance indicators drawn from government datasets with qualitative ‘signals’ focused on drivers of change within the system. Assessment of signals in turn required synthesizing different sources of information, including informal channels of obtaining evidence such as partner meetings or government-sponsored events. The evaluating team thus consciously empowered different kinds of researchers with differential expertise with the goal of building a much more pixelated picture. The goal was not to identify a single source of truth but rather a patchwork of validated information where the relevance of different pieces of data were dependent on evolving outcomes of interest.   The second set of competencies related to the skills required to play a convening role for donors and implementing partners, supporting better understanding of the changing operating context and help inform decision-making by program officers and partners. This involved building and sustaining relationships across different stakeholders at different stages of the project – from proposal development to review. Competencies relating to effective dialogue and developing an understanding of the core interests of international, national and sub-national partners as well as international donors and experts could only be developed iteratively and over time, but this was crucial in a distributed health decision-making ecosystem like India. The third and final set of competencies relate to operational adaptiveness, while evaluating an ecosystem with few constants. This can be the hardest competency to acquire because it is the farthest from the traditional notions embedded in the training of evaluation scientists and MERL practitioners. We found that the degree of buy-in and commitment to previously agreed upon frames of reference for evaluation can be shifted by changes in personnel or internal organizational structures. These shifts can lead to chain reactions of mismatched expectations that needed to be understood and managed in real time by MERL partners. The pandemic further created a natural experiment that on the one hand required a reexamination of program priorities and on the other depended on reliability of donor support. Each of these three types of capacities – synthesizing nuanced evidence for adaptive action, relationship building and communication, and managing operational discontinuities are in fact inter-dependent. Building evaluator competencies isn’t simply about capacity-building but rather a recognition of the diversity of skills and worldviews that need to be encompassed within our monitoring and evaluation functions for today’s complex, discontinuous health systems.