Using pre- and post-survey instruments in interventions: determining the random response benchmark and its implications for measuring effectiveness

Abstract

AbstractObjectiveThe present communication demonstrates that even if individuals are answering a pre/post survey at random, the percentage of individuals showing improvement from the pre- to the post-survey can be surprisingly high. Some simple formulas and tables are presented that will allow analysts to quickly determine the expected percentage of individuals showing improvement if participants just answered the survey at random. This benchmark percentage, in turn, defines the appropriate null hypothesis for testing if the actual percentage observed is greater than the expected random answering percentage.DesignThe analysis is demonstrated by testing if actual improvement in a component of the US Department of Agriculture’s (USDA) Expanded Food and Nutrition Education Program is significantly different from random answering improvement.SettingUSA.SubjectsFrom 2011 to 2014, 364320 adults completed a standardized pre- and post-survey administered by the USDA.ResultsFor each year, the statement that the actual number of improvements is less than the expected number if the questions were just answered at random cannot be rejected. This does not mean that the pre-/post-test survey instrument is flawed, only that the data are being inappropriately evaluated.ConclusionsKnowing the percentage of individuals showing improvement on a pre/post survey instrument when questions are randomly answered is an important benchmark number to determine in order to draw valid inferences about nutrition interventions. The results presented here should help analysts in determining this benchmark number for some common survey structures and avoid drawing faulty inferences about the effectiveness of an intervention.