Empirical Performance of Optimal Bayesian Adaptive Estimation

Abstract

Simulation studies have shown how Bayesian adaptive estimation methods should be set up for optimal performance. We assessed the extent to which these results hold up for human observers, who are more subject to failure than simulation subjects. Discrimination and detection experiments with two-alternative forced-choice (2AFC) tasks were used for that purpose. Forty estimates of the point of subjective equality (PSE, or the 50% correct point on the psychometric function for discrimination) and 32 estimates of detection threshold (the 80% correct point on the psychometric function for detection) were taken for each of four observers with the optimal Bayesian method, while data for fitting the psychometric function Ψ were gathered concurrently with an adaptive method of constant stimuli governed by fixed-step-size staircases. The estimated parameters of the psychometric function served as a criterion for comparison. In the discrimination task, PSEs for each observer were distributed around the independently estimated 50% correct point on Ψ and their variability was occasionally minimally larger than simulation results indicated it should be. In the detection task, the distribution of threshold estimates was consistently above the independently estimated 80% correct point on Ψ and their variability was as expected from simulations. A close analysis of these results suggests that the optimal Bayesian method is affected by growing inattention or fatigue in detection tasks (factors that are not considered in simulations), and limits the practical applicability of Bayesian estimation of detection thresholds.