Using causal models to show the effect of untestable assumptions on effect estimates in randomized controlled trials

Abstract

Background The methods by which randomized controlled trials (RCTs) are analyzed rest on several assumptions, most of which are untestable. Purpose To show how estimates of the net effect of treatment on survival can be obtained requiring only the assumption that randomization produced equivalent groups. Methods The assumptions underlying ratio measures of effect, based on disease occurrence times (DOT) obtained from survival curves, are identified and cumulatively removed. Results The four assumptions usually made are that (1) the ratio of disease incidence rates under treatment and under reference exposure is constant over time, (2) the groups being compared are exchangeable (equivalent), (3) a subject's DOT under treatment is independent of what his DOT would have been under the reference exposure, and (4) the treatment effect, if any, is in the same direction in all subjects. Removing Assumption 4 leads to an estimator of effect resembling the etiologic fraction, but able to accommodate both causative and preventive effects. Removing all assumptions but that of exchangeability still permits the estimation, directly from the survival curves, of a range of effect magnitudes, causative or preventive, compatible with the observed DOTs. The exchangeability assumption is the easiest to meet, by randomizing enough subjects. Limitations The statistical uncertainty that affects the estimates of survival probabilities has been ignored. Taking uncertainty into account further widens the range of effects compatible with the observations. Conclusions Retaining only the exchangeability assumption allows for a range of possible treatment effects to be estimated, although it may be wide. Readers of RCT reports should understand that the determination of a point estimate of effect within this range is entirely a function of unverifiable analytic assumptions. Clinical Trials 2007; 4: 611—620. http://ctj.sagepub.com