Performance Validity Testing on the NIH Toolbox Cognition Battery: Base Rates of Failed Embedded Validity Indicators in the Adult Normative Sample

Abstract

Abstract Objective The goal of this study was to determine the base rates of failing proposed embedded validity indicators (EVIs) for the National Institutes of Health Toolbox Cognition Battery (NIHTB-CB) in the normative sample. Method Participants included adults in the NIHTB-CB normative sample with data to calculate age-adjusted standard scores (n = 855; ages: M(SD) = 46.9(17.3), range: 18–85; 65.0% women; education: M(SD) = 14.1(2.5) years) or demographically adjusted T-scores (n = 803; ages: M(SD) = 47.3(17.3), range: 18–85; 65.3% women; education: M(SD) = 14.2(2.5) years) for all tests. The NIHTB-CB includes two tests of crystallized cognition and five tests of fluid cognition. Individual norm-referenced test performances were categorized as falling above or below liberal and conservative cutoffs based on proposed univariate EVIs. The number of univariate EVI failures was summed to compute multivariable EVIs. EVI failure rates above 10% were considered high false-positive rates, indicating specificity < .90. Using chi-square analyses, the frequencies of EVI failures were compared based on gender, race/ethnicity, education, and crystallized composite. Results The multivariable EVIs had predominantly low false-positive rates in the normative sample. EVI failure rates were most common among participants with low crystallized composites. Using age-adjusted standard scores, EVI failure rates varied by education, race/ethnicity, and estimated premorbid intelligence. These differences were mostly eliminated when using demographically adjusted T-scores. Conclusions Multivariable EVIs requiring ≥ 4 failures using liberal cutoffs or ≥ 3 failures using conservative cutoffs had acceptable false-positive rates (i.e., < 10%) using both age-adjusted standard scores and demographically adjusted T-scores. These multivariable EVIs could be applied to large data sets with NIHTB-CB data to screen for potentially invalid test performances.