Development and Interrater Reliability of an Observational School Environment Checklist: A Practical, Comprehensive Tool to Assess Healthy Eating and Physical Activity Opportunities in Schools

Abstract

Introduction Comprehensive, objective assessment of schools’ eating and physical activity environments is critical to developing and evaluating policies and interventions to reduce pediatric obesity inequities; however, few tools exist that describe the entire school comprehensively and are feasible with restricted resources. This study describes development and reliability of the observational school environment checklist (OSEC), a comprehensive observational audit tool. Method We developed the OSEC through iterative adaptations of existing instruments and pilot testing. The tool assesses four focus areas: cafeteria, lobby/hallway, gym, and outdoor areas. For reliability testing, two trained auditors independently completed the OSEC and met to resolve disagreements. For items with poor agreement, a third independent coder coded photographs taken during auditing. Percent agreement and Cohen’s kappa were calculated for all items and across four evidence-based constructs: atmosphere, accessibility, attractiveness, and advertising. Results After iterative development, the 88-item OSEC was tested for reliability in 18 schools. Items with poor (<80%) agreement or redundancy were discarded or reworded ( n = 16 items). All four constructs had acceptable agreement, ranging by focus area: 72.3% (attractiveness), 86.3% to 97.1% (atmosphere), 82.9% to 100% (accessibility), and 92.9% (advertising). Cohen’s kappa ranges were acceptable: 0.66–0.91 (atmosphere), 0.60–1.00 (accessibility), 0.46 (attractiveness), and 0.77 (advertising). After adding similar items across domains ( n = 49) to improve comprehensiveness, the final tool contained 121 binary items. Implications The OSEC reliably and comprehensively captures the school environment. It requires few resources or expertise to administer, has acceptable reliability, and can assess atmosphere, accessibility, attractiveness, and advertising in school areas where students engage in eating and physical activity.