Evaluating analytic scoring for the TOEFL® Academic Speaking Test (TAST) for operational use

Abstract

This study explores the utility of analytic scoring for TAST in providing useful and reliable diagnostic information for operational use in three aspects of candidates' performance: delivery, language use and topic development. One hundred and forty examinees' responses to six TAST tasks were scored analytically on these three aspects of speech. G studies were used to investigate the dependability of the analytic scores, the distinctness of the analytic dimensions, and the variability of analytic score profiles. Raters' perceptions of dimension separability were obtained using a questionnaire. It was found that the dependability of analytic scores averaged across six tasks and double ratings was acceptable for both operational and practice settings. However, scores averaged across two tasks and double ratings were not reliable enough for operational use. Correlations among the analytic scores by task were high but those between delivery and topic development were lower. These results were corroborated by raters' perceptions. When averaged across tasks or task types, correlations among the analytic scores were very high, and the profiles of scores were flat. The utility of analytic scoring is discussed, considering both score dependability and whether analytic scores provide diagnostic information beyond that provided by holistic scores.