A Practical Solution to Optimizing the Reliability of Teaching Observation Measures Under Budget Constraints

Abstract

Researchers often use generalizability theory to estimate relative error variance and reliability in teaching observation measures. They also use it to plan future studies and design the best possible measurement procedures. However, designing the best possible measurement procedure comes at a cost, and researchers must stay within their budget when designing a study. In this study, we applied the LaGrange multiplier method to obtain facet sample size equations that minimize relative error variance (hence maximize reliability) under budget constraints. We did this for a crossed design and three nested designs that are more typical of data collection in teaching observation studies. Using an example budget and variance components similar to those found in practice, we demonstrate the use of these equations. We also show the way variance components from fully crossed designs can be combined to use our equations for a nested design.