Peer-Reviewed Validation of a Comprehensive Framework for Disaster Evaluation Typologies

Abstract

AbstractIntroduction:The Comprehensive Framework for Disaster Evaluation Typologies, developed in 2017 (CFDET 2017), aims to unify and facilitate agreement regarding the identification, structure, and relationships between various evaluation typologies found in the disaster setting. A peer-reviewed validation process sought input from international experts in the fields of disaster medicine, disaster/emergency management, humanitarian/development, and evaluation. This paper discusses the validation process, its results, and outcomes.Research Problem:Previous frameworks, identified in the literature, lack validation and consistent terminology. To gain credibility and utility, this unique framework needed to be validated by international experts in the disaster setting.Methods:A mixed methods approach was designed to validate the framework. An initial iterative process informed an online survey which used a combination of a five-point Likert scale and open-ended questions. Pre-determined consensus thresholds, informed by a targeted literature review, provided the validation criteria.Results:A sample of 33 experts from 11 countries responded to the validation process. Quantitative measures largely supported the elements and relationships of the framework, and strongly supported its value and usefulness for supporting, promoting, and undertaking evaluations, as well as its usefulness for teaching evaluation in the disaster setting. Qualitative input suggested opportunities to strengthen and enhance the framework. There were limited responses to better understand the barriers and enablers of undertaking disaster evaluations. A potential for self-selection bias of respondents may be a limitation of this study. The attainment of high consensus thresholds, however, provides confidence in the validity of the results.Conclusion:For the first time, a framework of this nature has undergone a rigorous validation process by experts in three related disciplines at an international level. The modified framework, CFDET 2018, provides a unifying framework within which existing evaluation typologies can be structured. It gives evaluators confidence to choose an appropriate strategy for their particular evaluation in the disaster setting and facilitates consistency in reporting across the different phases of a disaster to better understand the process, outcomes, and impacts of the efficacy and efficiency of interventions. Future research could create a series of toolkits to support improved disaster evaluation processes and to evaluate the utility of the framework in the real-world setting.