A simulation-based method for model evaluation

Abstract

We wish to evaluate and compare models that are non-nested and fit to data using different fitting criteria. We first estimate parameters in all models by optimizing goodness-of-fit to a dataset. Then, to assess a candidate model, we simulate a population of datasets from it and evaluate the goodness-of-fit of all the models, without re-estimating parameter values. Finally, we see whether the vector of goodness-of-fit criteria for the original data is compatible with the multivariate distribution of these criteria for the simulated datasets. By simulating from each model in turn, we determine whether any, or several, models are consistent with the data. We apply the method to compare three models, fit at different temporal resolutions to binary time series of animal behaviour data, concluding that a semi-Markov model gives a better fit than latent Gaussian and hidden Markov models.