Get REAL: Development and Validation of the Rubric for the Evaluation of Apps for Learning

Abstract

Purpose: Technology, including educational applications (apps), is commonly used in schools by teachers and speech-language pathologists. Nonetheless, very little research has examined the efficacy of these apps for student learning or how to choose appropriate apps for instruction. Several previous rubrics to evaluate the instructional quality of apps have been published, yet limitations such as user training and time to administer have limited their use. The purpose of this study was to develop a comprehensive yet easy-to-use rubric for evaluating educational apps. Method: An iterative process guided the development and assessment of the Rubric for the Evaluation of Apps for Learning (REAL). Prior rubrics were synthesized to develop a comprehensive list of items. These items were condensed for simplicity purposes. A focus group and a survey of teachers and speech-language pathologists validated the rubric items and informed revisions. Multiple rounds of interrater reliability calculations and rubric revisions resulted in a final draft of the rubric, included here. Results: The final version of REAL passed validation and demonstrated evidence of interrater reliability among familiar users (Fleiss' κ = .71–.77) and those with no prior experience using the rubric (Fleiss' κ = .46). The majority of practitioners (90%) reported that they would use the rubric in their practice. REAL was used to rate 10 apps teaching phonics and 10 apps teaching social skills; scores are presented for demonstration purposes and indicate a wide range of rankings. Conclusions: REAL is a feasible, comprehensive rubric for evaluating the quality of educational apps, for which there may be little to no published research evidence. Practitioners may be overwhelmed with the number of commercially available apps. REAL provides a tool to help them make decisions that benefit the children they serve.