Measuring Tornado Warning Reception, Comprehension, and Response in the United States

Abstract

Abstract Social criteria are important to achieving the mission of the National Weather Service. Accordingly, researchers and administrators at the NWS increasingly recognize a need to supplement verification statistics with complementary data about society in performance management and evaluation. This will require significant development of new capacities to both conceptualize relevant criteria and measure them using consistent, transparent, replicable, and reliable measures that permit generalizable inference to populations of interest. In this study, we contribute to this development by suggesting three criteria that require measurement (forecast and warning reception, comprehension, and response) and demonstrating a methodology that allows us to measure these concepts in a single information domain—tornado warnings. The methodology we employ improves upon previous research in multiple ways. It provides a more generalizable approach to measurement using a temporally consistent set of survey questions that are applicable across the United States; it relies on a more robust set of psychometric tests to analytically demonstrate the reliability of the measures; and it is more transparent and replicable than previous research because the data and methods (source code) are publicly available. In addition to describing and assessing the reliability of the measures, we explore the sensitivity of the measures to geographic and demographic variation to identify significant differences that require attention in measurement. We close by discussing the implications of this study and the next steps toward development and use of social criteria in performance management and evaluation.