Author:
Huang Jinyan,Whipple Patrick B.
Abstract
AbstractUsing generalizability (G-) theory as a theoretical framework and research methodology, this study examined the impact of the current one-rater holistic scoring practice on the rater variability and reliability of constructed response questions in New York State high-stakes tests of grades four and six English language arts (ELA) and grades four and five mathematics assessments. Following the New York State scoring rubrics, a total of 36 grades four and six ELA constructed response samples and 72 grades four and five mathematics constructed response samples were marked holistically by ten independent raters having current certifications as educators in the State of New York. The results indicated that the current single-rater holistic scoring practice would not be able to yield acceptable G-coefficients for the New York State grades four and six ELA and grades four and five mathematics assessments. Implications for assessment policy making at the local and state levels are discussed.
Publisher
Springer Science and Business Media LLC
Subject
General Economics, Econometrics and Finance,General Psychology,General Social Sciences,General Arts and Humanities,General Business, Management and Accounting
Reference54 articles.
1. American Educational Research Association (AERA), American Psychological Association (APA), and National Council on Measurement in Education (NCME) (2014) Standards Edu Psychol Testing. American Psychological Association, Washington, DC
2. Andrade HL, Du Y, Wang X (2008) Putting rubrics to the test: The effect of a model, criteria generation, and rubric-referenced self-assessment on elementary school students’ writing. Edu Measurement: Issues Practice 27(2):3–13
3. Bacha N (2001) Writing evaluation: what can analytic versus holistic essay scoring tell us? System 29(3):371–383
4. Badjadi, N. E. I. (2013). Conceptualizing essay tests’ reliability and validity: From research to theory. Online Submission
5. Barkaoui K (2011) Effects of marking method and rater experience on ESL essay scores and rater performance. Ass Edu: Principles, Policy & Practice 18(3):279–293