Detecting a Change in School Performance: A Bayesian Analysis for a Multilevel Join Point Problem

Abstract

A substantial literature on switches in linear regression functions considers situations in which the regression function is discontinuous at an unknown value of the regressor, Xk , where k is the so-called unknown “change point.” The regression model is thus a two-phase composite of yi ∼ N(β01 + β11xi, σ12), i=1, 2,..., k and yi ∼ N(β02 + β12xi, σ22), i= k + 1, k + 2,..., n. Solutions to this single series problem are considerably more complex when we consider a wrinkle frequently encountered in evaluation studies of system interventions, in that a system typically comprises multiple members (j = 1, 2, . . . , m ) and that members of the system cannot all be expected to change synchronously. For example, schools differ not only in whether a program, implemented system-wide, improves their students’ test scores, but depending on the resources already in place, schools may also differ in when they start to show effects of the program. If ignored, heterogeneity among schools in when the program takes initial effect undermines any program evaluation that assumes that change points are known and that they are the same for all schools. To describe individual behavior within a system better, and using a sample of longitudinal test scores from a large urban school system, we consider hierarchical Bayes estimation of a multilevel linear regression model in which each individual regression slope of test score on time switches at some unknown point in time, kj. We further explore additional results employing models that accommodate case weights and shorter time series.