Better Than Maximum Likelihood Estimation of Model- based and Model-free Learning Styles

Abstract

Abstract Various decision-making systems work together to shape human behavior. Goal-directed and habitual systems are the two most important systems studied by reinforcement learning (RL) through model-based (MB) and model-free (MF) learning styles, respectively. Human behavior resembles the combination of these two decision-making paradigms, achieved by the weighted sum of the action values of the two styles in an RL framework. The weighting parameter is often extracted by the maximum likelihood (ML) or maximum a-posteriori (MAP) estimation method. In this study, we employ RL agents that use a combination of MB and MF decision-making to perform the well-known Daw two-stage task. ML and MAP methods result in less reliable estimates of the weighting parameter, where a large bias toward extreme values is often observed. We propose the knearest neighbor as an alternative nonparametric estimate to improve the estimation error, where we devise a set of 20 features extracted from the behavior of the RL agent. Simulated experiments examine the proposed method. Our method reduces the bias and variance of the estimation error based on the obtained results. Human behavior data from previous studies is investigated as well. The proposed method results in predicting indices such as age, gender, IQ, the dwell time of gaze, and psychiatric disorder indices which are missed by the traditional method. In brief, the proposed method increases the reliability of the estimated parameters and enhances the applicability of reinforcement learning paradigms in clinical trials.