Simulating markov-reward processes with rare events

Abstract

Simulating continuous-time Markov reward processes containing rarely visited, but economically important states requires long simulation times unless special measures are taken. In this article, we consider Markov reward processes in equilibrium, and we use the equilibrium equations to reallocate rewards. The effect of this reallocation is determined analytically for a number of small examples. In these examples, significant savings in run lengths were possible, especially in the case where the expected rewards are strongly influenced by low-probability boundary states. The emphasis of the article is on exploration; no large simulation problems have been considered.