Using Qualitative Data for Estimating Construct Validity of Standardized Measures

Abstract

This paper describes efforts to develop a method of using qualitative interview data for investigating the construct validity of standardized measures. Illustration is provided through a multimethod validation study that examined the concurrence between observer assessments of interview data and a standardized paper and pencil measure–the Demands of Immigration Scale (DI). Specific aims of the multimethod validation study included (1) determining interrater reliability of the Demands of Immigration Observer Rating Scale (DI-ORS), which assessed respondents according to the interview data they provided and (2) establishing concurrence between observer assessments on the DI-ORS and the respondents’ self-ratings on the DI scale. Interrater reliability of the DI-ORS was achieved with 97.9% agreement. However, there were significant differences (p ≤ .05,2-tailed) or disagreement on 5 out of 6 possible occurrences for agreement or disagreement between observer ratings on the DI-ORS and respondents’ self ratings on the DI scale. The findings are informative for researchers who wish to use qualitative methods for construct validation.