Re-weighting the randomized controlled trial for generalization: finite-sample error and variable selection

Abstract

Abstract Randomized controlled trials (RCTs) may suffer from limited scope. In particular, samples may be unrepresentative: some RCTs over- or under-sample individuals with certain characteristics compared to the target population, for which one wants conclusions on treatment effectiveness. Re-weighting trial individuals to match the target population can improve the treatment effect estimation. In this work, we establish the expressions of the bias and variance of such re-weighting procedures—also called inverse propensity of sampling weighting (IPSW)—in presence of categorical covariates for any sample size. Such results allow us to compare the theoretical performance of different versions of IPSW estimates. Besides, our results show how the performance (bias, variance, and quadratic risk) of IPSW estimates depends on the two sample sizes (RCT and target population). A by-product of our work is the proof of consistency of IPSW estimates. In addition, we analyse how including covariates that are not necessary for identifiability of the causal effect may impact the asymptotic variance. Including covariates that are shifted between the two samples but not treatment-effect modifiers increases the variance while non-shifted but treatment-effect modifiers do not. We illustrate all the takeaways in a didactic example, and on a semi-synthetic simulation inspired from critical care medicine.