The Complexity of Measuring the Quality of Program Implementation With Observations

Abstract

In the growing literature on the evaluation of program implementation, less has been said about evaluating program quality than about evaluating other aspects of program implementation. Furthermore, most articles and reports in the program-implementation evaluation literature have presented only brief descriptions of how implementation instruments have been developed. In this article, the authors describe a method for evaluating the quality of implementation of middle school inquiry-based science using data from observations scaled with paired comparison judgments. The authors show the complexities of developing and applying the method, describe how they tried it out, present the results of validity and reliability analyses, and describe the method's strengths and weaknesses.