Decoupling power and type I error rate considerations when incorporating historical control data using a test‐then‐pool approach

Abstract

AbstractTo accelerate a randomized controlled trial, historical control data may be used after ensuring little heterogeneity between the historical and current trials. The test‐then‐pool approach is a simple frequentist borrowing method that assesses the similarity between historical and current control data using a two‐sided test. A limitation of the conventional test‐then‐pool method is the inability to control the type I error rate and power for the primary hypothesis separately and flexibly for heterogeneity between trials. This is because the two‐sided test focuses on the absolute value of the mean difference between the historical and current controls. In this paper, we propose a new test‐then‐pool method that splits the two‐sided hypothesis of the conventional method into two one‐sided hypotheses. Testing each one‐sided hypothesis with different significance levels allows for the separate control of the type I error rate and power for heterogeneity between trials. We also propose a significance‐level selection approach based on the maximum type I error rate and the minimum power. The proposed method prevented a decrease in power even when there was heterogeneity between trials while controlling type I error at a maximum tolerable type I error rate larger than the targeted type I error rate. The application of depression trial data and hypothetical trial data further supported the usefulness of the proposed method.