Evaluator’s alignment as an important indicator of adequacy of the criteria and assessment procedure for recognizing the good practice in public health

Abstract

BackgroundPublic health interventions aim to reduce the burden of chronic non-communicable diseases. Implementing evidence-based interventions that are proven to be successful and effective is widely recognized as the best approach to addressing public health challenges. To avoid the development and implementation of less effective or successful or even harmful practices, clear criteria for the assessment of practices, that consider different dimensions of the interventions in public health, are needed. The main aim of the research was to test our Criteria and assessment procedure for recognizing good practices in the field of public health by estimating the consistency between the evaluators and thereby gaining insight into the adequacy and reliability of the criteria as well as to check how the evaluators understand the criteria and methodology and if it is properly used in assessing the interventions.MethodsThe assessment of the interventions took place from 2021 to 2022. The individual evaluator’s scores on the scale from 1 to 5 for each specific sub-criterion were collected, which was followed by a panel discussion to reach a final score for each sub-criterion. The inter-rater agreement was measured using percent overall agreement and Fleiss’ kappa coefficient.ResultsWe found moderate inter-rater agreement on the level of the assessment criteria group. The lowest agreement was observed for the effectiveness and efficiency sub-criteria group, which also received the lowest scores from the evaluators. Challenges identified with the scoring process were due to the descriptive 1 to 5 scale and the varying specificity of the criteria.ConclusionThe results showed that studying consistency between evaluators can highlight areas for improvement or adjustment in the assessment criteria and enhance the quality of the assessment instrument. Therefore, such analysis would be useful part of both newly and well-established health promotion and prevention program registries.