Applications of Markov Chain Approximation Methods to Optimal Control Problems in Economics

Abstract

In this paper we explore some benefits of using the finite-state Markov chain approximation (MCA) method of Kushner and Dupuis (2001) to solve continuous-time optimal control problems in economics. We first show that the implicit finite-difference scheme of Achdou et al. (2022) amounts to a limiting form of the MCA method for a certain choice of approximating chains and policy function iteration for the resulting system of equations. We then illustrate that, relative to the implicit finite-difference approach, using variations of modified policy function iteration to solve income fluctuation problems both with and without discrete choices can lead to an increase in the speed of convergence of more than an order of magnitude. Finally, we provide several consistent chain constructions for stationary portfolio problems with correlated state variables, and illustrate the flexibility of the MCA approach by using it to construct and compare two simple solution methods for a general equilibrium model with financial frictions. Replication materials may be found at https://github.com/tphelanECON/EslamiPhelan_MCA.