Estimating Reliabilities of Computerized Adaptive Tests

Abstract

This paper presents two methods for estimating the reliability of a computerized adaptive test (CAT) with out using item response theory. The required data con sist of CAT and paper-pencil (PP) scores from identical or equivalent samples, and scores for all examinees on one or more covariates. Multiple R 2s and communali ties are used to compute the ratio of CAT and PP relia bilities. When combined with the PP reliability calcu lated by a conventional procedure, these ratios yield estimates of CAT reliability. Index terms: computer ized adaptive testing, item response theory, predictive validity, reliability, tailored testing.