Proceed with Caution: Interactive Rules and Teacher Work Sample Scoring Strategies, an Ethnomethodological Study

Abstract

Background Facing growing accountability pressures, Teacher Work Samples (TWSs) as a model of performance-based assessment is of growing significance in teacher education. Developed at Western Oregon University and widely adopted and adapted, proponents claim that the model is “real,” “natural,” “meaningful,” and “helpful” (G. R. Girod, 2002). Research Questions The study addresses three questions: (1) How do sample raters understand their responsibilities? (2) What are the underlying interactive rules and strategies used by raters to achieve their aims, and how are they employed? (3) What issues or concerns should teacher educators interested in using TWS methods address as they seek to demonstrate candidate quality and program value? Research Design Conversation analysis, ethnomethodology. Data Collection and Analysis Ten TWS scoring conversations conducted by four teams were recorded and analyzed to identify interactive rules and strategies. Scoring teams were composed of one tenure-track elementary teacher education faculty member and one clinical teacher education faculty member. Excerpts from a TWS case judged marginal are presented and analyzed. Findings From the case, a set of interactive rules (tenure-track faculty speak first; the efficiency and equivalence rules; and scorers are prepared) and strategies (splitting the difference; rubric simplification; previewing scores; and rubric stretching) are identified, and implications of their use are discussed for assessment validity, fairness, content quality and coverage, meaningfulness, and cognitive complexity (R. L. Linn, E. L. Baker, & S. B. Dunbar, 1991). Conclusions This study raises a number of concerns about the expectations for and use of Teacher Work Samples and cautions about their use for high-stakes assessment.