Affiliation:
1. Department of Statistics, Rutgers University , Piscataway , United States
2. Department of Statistics, Department of Computer Science, Columbia Data Science Institute, Columbia University , New York , United States
3. Department of Computer Science, Center for Data Science, New York University , New York , United States
Abstract
AbstractBayesian modelling helps applied researchers to articulate assumptions about their data and develop models tailored for specific applications. Thanks to good methods for approximate posterior inference, researchers can now easily build, use, and revise complicated Bayesian models for large and rich data. These capabilities, however, bring into focus the problem of model criticism. Researchers need tools to diagnose the fitness of their models, to understand where they fall short, and to guide their revision. In this paper, we develop a new method for Bayesian model criticism, the holdout predictive check (HPC). Holdout predictive check are built on posterior predictive check (PPC), a seminal method that checks a model by assessing the posterior predictive distribution on the observed data. However, PPC use the data twice—both to calculate the posterior predictive and to evaluate it—which can lead to uncalibrated p-values. Holdout predictive check, in contrast, compare the posterior predictive distribution to a draw from the population distribution, a heldout dataset. This method blends Bayesian modelling with frequentist assessment. Unlike the PPC, we prove that the HPC is properly calibrated. Empirically, we study HPC on classical regression, a hierarchical model of text data, and factor analysis.
Funder
Simons Foundation
Sloan Foundation
Open Philanthropy
NIH
NHLBI
Eric and Wendy Schmidt Center
Publisher
Oxford University Press (OUP)
Subject
Statistics, Probability and Uncertainty,Statistics and Probability
Reference37 articles.
1. P-values for composite null models;Bayarri;Journal of the American Statistical Association,2000
2. Bayesian checking of the second levels of hierarchical models;Bayarri;Statistical Science,2007
3. Bayesian measures of surprise for outlier detection;Bayarri;Journal of Statistical Planning and Inference,2003
4. The intrinsic Bayes factor for model selection and prediction;Berger;Journal of the American Statistical Association,1996
5. Build, compute, critique, repeat: Data analysis with latent variable models;Blei;Annual Review of Statistics and its Application,2014