On the interplay between practical and statistical significance in equal employment cases

Abstract

Abstract The distinction between statistical and practical significance often arises in the analysis of large data sets, where seemingly small disparities can reach statistical significance at the 0.05 level. In 2014, the opinion in Jones v. Boston created a split in the circuits when it relied solely on statistical significance. The opinion noted that there is no generally accepted definition of practical significance and that the phrase may simply mean that the person using the term views the disparity as substantial enough that a plaintiff should be able to sue. In July 2019, the Office of Federal Compliance Programs (OFCCP) proposed updated procedures for the way it considers both practical and statistical significance when reviewing the employment practices of government contractors.1 On 10 November 2020, the Agency issued its final rules, which downplayed the role of statistical significance.2 The main measures of practical significance mentioned in the notices OFCCP issued in the Federal Register or its Frequently Asked Questions guide3 as well as some other measures of practical significance that have been proposed are studied in this article. The results of the paper support the First Circuit and OFCCP in their views that there is no single measure of practical significance that can be recommended for use as the sole criterion for deciding whether the disparity has practical importance. Nevertheless, the paper includes an empirical guideline combining the odds ratio and difference between success rates to judge practical significance as well as recommendations for balancing Type I and Type II errors and their consequences in the context of the case in determining the level required for statistical significance.