Complexity of Illustrations in PISA-2009 Science Items and Its Relationship to the Performance of Students from Shanghai-China, the United States, and Mexico

Abstract

Background While illustrations are widely used in international test comparisons, very scant research has been conducted on their design and on their influence on student performance. It is not clear how the features of illustration act in combination supporting students’ access to the content of items or increasing their interpretation demands. Purpose The purpose of this study was to examine how the characteristics of illustrations used in science items are related to the performance of students in PISA-2009. Population We used PISA-2009 data on the responses of 1571 to 1582, 1601 to 1617, and 11,662 to 11,746 students, respectively from Shanghai-China, the U.S., and Mexico. Research Design This was a secondary analysis study that compared the three PISA-2009 jurisdictions as to the magnitudes and directions of the correlations between the complexity of illustrations used in science items and the difficulty of those items. We used the information on item difficulty provided by the Organisation for Economic Co-operation and Development (OECD, 2010). The features of the illustrations were coded with a system that coded the presence or absence of about 100 types of illustration features. The complexity of each illustration was computed as the number of different types of features observed. Findings For the U.S. and Mexico, items with higher illustration complexities were more difficult than items with lower illustration complexities. For Shanghai-China, the correlation of number of illustration features and item difficulty was close to zero. This pattern of correlations is consistent with the three jurisdictions’ ranking in PISA-2009. Conclusions While this was an exploratory study that examined the performance of only three of the 65 PISA-2009 jurisdictions, the results speak to the importance of examining illustration complexity as a factor that shapes student performance in science tests and which should be addressed systematically in the design of science items.