Generalizability Analyses of Work Keys Listening and Writing Tests

Abstract

In this article, univariate and multivariate generalizability theory is employed to examine psychometric characteristics of Listening and Writing tests developed by American College Testing (ACT) for its Work Keys program. For both of these tests, the same tape-recorded messages provide the stimuli for written responses that are evaluated by raters. Of particular concern are the numbers of messages and raters needed for adequate measurement precision. The occurrence of relatively large person-message interactions suggested that several (at least six) messages are probably required. Also, the analyses suggest that it is highly desirable to employ at least two raters. For these sample sizes, however, the multivariate results are less encouraging in part because of correlated errors.