Election polls are 95% confident but only 60% accurate

Abstract

Election polls in the United States are more confident than accurate-meaning the reported margins of error often do not encompass the actual election outcomes in spite of pollsters claiming a 95% confidence level (that is, a 95% chance that their predictions will fall within the margin of error). In an analysis of polls for more than 6,000 contests, we have found that the actual vote total for a given candidate fell within the 95% confidence interval for just 60% of the polls. This degree of accuracy was reached only when the polls were conducted in the week before an election; accuracy was worse for polls conducted earlier. Polls would, in fact, need margins of error at least twice their current standard reported width to achieve 95% accuracy. We have also found that when laypeople read about poll results, they tend to overestimate the poll's accuracy, even when they have historical data demonstrating that the predictions made by polls are often inaccurate. These results illustrate polls’ vulnerability to overconfidence and the limitations of the lay public's understanding of these shortcomings. We conclude by suggesting ways that pollsters and reporters could enable the public to interpret poll data more realistically.