How Many Classes and Students Should Ideally be Sampled When Assessing the Role of Classroom Climate via Student Ratings on a Limited Budget? An Optimal Design Perspective

Abstract

AbstractA central question in educational research is how classroom climate variables, such as teaching quality, goal structures, or interpersonal teacher behavior, are related to critical student outcomes, such as students’ achievement and motivation. Student ratings are frequently used to measure classroom climate. When using student ratings to assess classroom climate, researchers first ask students to rate classroom climate characteristics and then aggregate the ratings on the class level. Multilevel latent variable modeling is then used to determine whether class-mean ratings of classroom climate are predictive of student outcomes and to correct for unreliability so that the relations can be estimated without bias. In this article, we adopt an optimal design perspective on this specific strategy. Specifically, after briefly recapping a prominent model in climate research, we show and explain (a) how statistical power can be maximized by choosing optimal numbers of classes and students per class given a fixed budget for conducting a study and (b) how the budget required to achieve a prespecified level of power can be minimized. Moreover, we present an example from research on teaching quality to illustrate the procedures and to provide guidance to researchers who are interested in studying the role of classroom climate. Also, we present a Shiny App that can be used to help find optimal designs for classroom climate studies. The app can be accessed at https://psychtools.shinyapps.io/optimalDesignsClassroomClimate