Quantifying deliberate practice using auto‐graded questions: Analyzing multiple metrics in a chemical engineering course

Abstract

AbstractInteractive digital technologies provide a mechanism to capture big data for higher education faculty to monitor student progress in real‐time. By measuring both performance and perseverance using auto‐graded problems, this learning analytics research examines several tenets of deliberate practice. Specifically, online homework problems integrated into an interactive textbook for chemical engineering titled Material and Energy Balances zyBook were examined. Across three cohorts totaling over 250 students and 84,000 questions, the fraction correct, attempts before correct, and attempts after correct provided metrics to examine question type and order. Fraction correct and attempts before correct varied with statistical significance as a function of question type and order, including many differences having medium and large effect sizes. Multiple choice questions led to the highest fraction correct and the least number of attempts before correct, while questions requiring multiple numeric responses led to the lowest success and the largest number of attempts before correct. Next, multiple metrics were aggregated into a deliberate practice score, to broadly quantify problem difficulty. Fraction correct, modified correct, first quartile attempts before correct, and median attempts before correct were combined at different thresholds leading to a deliberate practice score between 0 and 8. Questions with higher deliberate practice scores were disproportionality used for practice (61% of attempts on 13% of questions); practice attempts do not receive course credit. Overall, combining metrics in a deliberate practice score exhibits the potential to optimize the length of homework assignments and give students more credit for solving more challenging auto‐graded problems.