Identifying Difficult Questions and Student Difficulties in a Spanish Version of a Programming Assessment Instrument (SCS1)

Abstract

Currently, there is little evidence about how non-English-speaking students learn computer programming. For example, there are few validated assessment instruments to measure the development of programming skills, especially for the Spanish-speaking population. Having valid assessment instruments is essential to identify the difficulties of the students, make formative decisions, and evaluate the impact of pedagogical interventions. This article presents the outcomes of the early steps toward a validation in Spanish of the SCS1 assessment, derived from the first conceptual inventory to measure programming knowledge decoupled from a particular language. The goal of this initial step is twofold: (1) identifying the questions that are too difficult in this version and might need to be adapted based on empirical evidence of how students answer them and (2) identifying students’ common difficulties with the most challenging questions. This study started with a “think-aloud” protocol to validate whether the Spanish version was similar to the original instrument and if the students understood the translation as intended. Then, the instrument was applied to a sample of 71 university students who had previously taken an introductory programming course using Python. We computed the difficulty index for each question and identified students’ difficulties in answering them. We also propose potential improvements for these items. The main contributions of this study include (1) progress toward the validation of an instrument for evaluating programming skills in Spanish, (2) noting the advantage that experienced students in Python have to complete the instrument, (3) identifying the tendency of students to translate the pseudo-code from Spanish into English, and (4) the identification of common mistakes, like misunderstanding the scope of local variables or the confusion between returning and printing.