Robust estimation of loss‐based measures of model performance under covariate shift

Abstract

AbstractWe present methods for estimating loss‐based measures of the performance of a prediction model in a target population that differs from the source population in which the model was developed, in settings where outcome and covariate data are available from the source population but only covariate data are available on a simple random sample from the target population. Prior work adjusting for differences between the two populations has used various weighting estimators with inverse odds or density ratio weights. Here, we develop more robust estimators for the target population risk (expected loss) that can be used with data‐adaptive (e.g., machine learning‐based) estimation of nuisance parameters. We examine the large‐sample properties of the estimators and evaluate finite‐sample performance in simulations. Last, we apply the methods to data from lung cancer screening using nationally representative data from the National Health and Nutrition Examination Survey (NHANES) and extend our methods to account for the complex survey design of the NHANES.