Assessing the Impact of Pediatric-Based Developmental Services on Infants, Families, and Clinicians: Challenges to Evaluating the Healthy Steps Program

Abstract

Background. Begun in 1996, the Healthy Steps for Young Children Program (HS) is a new model of pediatric practice that incorporates child development specialists and enhanced developmental services for families of young children. HS is for all families, not just those at high-risk. It is expected to strengthen parents' knowledge, attitudes, and behaviors in ways that promote child health and development, and in turn, to lead to improved child outcomes, such as improved language development, increased utilization of well child care, and decreased problem behaviors, hospitalizations, and injuries. The HS evaluation is designed to assess whether HS is successful in achieving the desired outcomes, measure the program's costs, and determine the relation of the program's costs to its outcomes. Objective. This article is the first report of the HS evaluation. It describes the evaluation design and characteristics of the HS sites and sample for the evaluation. Methods. The evaluation is following a cohort of children from birth to age 3 at 15 evaluation sites across the country. The sites represent a range of organizational practice settings that include group practices, hospital-based clinics, and health maintenance organization pediatric clinics. The evaluation design relies on 2 comparison strategies. At 6 randomization design sites, 400 children were randomized to the intervention or control group. At 9 quasi-experimental design sites, a comparison location with a similar organizational setting and patient profile has been selected and up to 200 children are being followed at each of these sites. At each site, 2 developmental specialists (or their full-time equivalents) work as a team with 4 to 8 pediatricians and pediatric nurse practitioners. The specialist conducts office visits (jointly or sequentially with the pediatric clinician) and home visits, assesses children's developmental progress, provides referrals and follow-up to resources in the community, organizes and conducts parent discussion groups, coordinates early reading activities, and maintains a telephone information line for questions about child development and behavior. The evaluation relies on many data sources including self-administered provider surveys, key informant interviews, forms completed by parents at office visits, telephone interviews with parents, medical record reviews, data from each site on program costs and health services use, and an ongoing log of family contacts maintained by each developmental specialist. Analyses for this article are based on enrollment data for the Healthy Steps sample and national data on 1997 US live births. The χ2 goodness-of-fit test was used to evaluate whether the distribution of selected demographic variables, insurance, and infant's birth weight for the Healthy Steps sample was similar to the distributions for US births in 1997. In addition, comparisons were made between intervention and comparison families at the randomization and quasi-experimental evaluation sites. The χ2 test of independence was used to evaluate differences in variables across groups. Results. Throughout a 26-month period, 5565 children enrolled in the evaluation, 2963 (53.2%) children in the intervention group and 2602 (46.8%) in the comparison group. More than 10% of mothers in the Healthy Steps sample are teenagers; 18% have 11 years of education or less; 27% have completed college; 18% are black or African-American; slightly >20% are of Hispanic origin; 36% are single; and close to one-third used Medicaid for their prenatal care. Approximately 7% of infants were low birth weight. When compared with national birth data for the United States as a whole, the Healthy Steps sample seems similarly diverse. However, with the exception of maternal age, the distribution of variables was significantly different from the distribution for US births. There are no differences between intervention and comparison families at randomization sites on any of the maternal characteristics, insurance status, or infant's birth weight. However, there are a number of differences between the intervention and the comparison groups at the quasi-experimental sites and between the randomization and quasi-experimental sites. At quasi-experimental sites, mothers in the comparison group were more likely than were mothers in the intervention group to be 29 years old or younger, to have fewer years of education, to be black or African-American, to report Hispanic origin, and to be single. Mothers at the quasi-experimental sites were slightly older than were mothers at randomization sites. They also were more likely to be married, to have <11 years of education (but also to be college graduates), to be of Hispanic origin, and to report private insurance or self-pay rather than Medicaid as their source of payment for prenatal care. Conclusion. The Healthy Steps sample is economically and ethnically diverse, reflecting the diversity of the nation as a whole. There seem to be differences between randomization and quasi-experimental sites as well as between intervention and comparison groups at the quasi-experimental design sites. As important, randomization at the sites that selected to use this approach seems to have been effective in equalizing the characteristics of families in the intervention and comparison groups. The differences between the intervention and comparison families at the quasi-experimental sites indicate the need to take account of these differences in analyzing program effects. In addition, to account for within-site correlation of outcomes, between-site variability of the effects of HS, and between-provider type variability of the effects of HS, random effects models will be used in the data analyses. These models are also referred to as hierarchical linear models or multilevel models. The HS evaluation is carefully designed to address the complexities of a program with multiple objectives, multiple components, and a wide range of expectations. The evaluation will provide information for practicing clinicians on the effectiveness of HS in improving care for families. It will provide policy makers with empirical evidence to inform the national debate on whether pediatric practices are an appropriate venue for helping parents promote their children's development. In addition, it will assess whether potential outcomes such as increased satisfaction with care and decreased hospitalizations make HS a valuable intervention.