A Low-Cost, Post Hoc Method to Rate Overall Site Quality in a Multi-Site Demonstration

Abstract

Demonstration programs and social experiments are often subject to sophisticated, controlled evaluations. An important factor that is not subject to control, and sometimes even goes unobserved, is overall program site quality. Site quality can be observed in process evaluations, but these tend to be expensive. This paper describes an alternative approach to site quality measurement. With observations elicited from national program staff of a large demonstration project, we developed an approach that uses Concept Mapping to identify dimensions for rating program site quality and compared our results with ratings based on site visits developed during program implementation. The site quality ratings and rankings resulting from our approach are plausible, have desirable psychometric properties, and are developed at much lower cost than is possible for even relatively modest process analyses.