How Close is Close Enough?

Abstract

Stagnant budgets and growing rates of Internet access have increased the appeal of replacing traditional survey methods with electronic ones. Online surveys are particularly appealing in the public transit industry because of the expense and logistical difficulty of surveying customers onboard buses and trains. It is therefore critical to understand, quantify, and test the differences between onboard versus online transit survey data. Traditional hypothesis tests are designed to show that two sample statistics most likely come from different populations. However, failing to find a difference cannot be interpreted as evidence that there is no difference. Furthermore, a difference may be statistically significant but so small as to provide no practical insight (which often happens when working with large sample sizes). Statistical equivalence testing provides an analytical framework with which to evaluate whether two data sets are similar enough to be interchangeable (i.e., statistically equivalent). The paper describes statistical equivalence tests conducted on customer satisfaction data collected onboard transit systems and data collected electronically with e-mail lists from users of the same systems. Researchers compared proportions of satisfied customers across various economic and travel behavior characteristics between these data sets. Within the chosen threshold of .05 (statistics within 5 percentage points of one another), one of the two data sets evaluated shows strong evidence of equivalence between onboard and online survey methods, while the other data set shows strong evidence of nonequivalence. Findings support the idea that, at least in some cases, online surveys can substitute for onboard ones.