On sets of countable non-negative matrices and Markov decision processes

Abstract

Consider a set S of countable non-negative matrices satisfying the property that for any two indices i, j, for some n ≧ 1 there are matrices M1, M2, · · ·, Mn in S with (M1M2 · · · Mn)ij >0. For non-negative vectors x set Tx = supM∈SMx, where the supremum is taken separately in each coordinate. Assume that for each x with Tx finite in each coordinate there is a matrix in S which achieves the supremum simultaneously for all coordinates. With these two assumptions on S, the R-theory for a countable irreducible matrix is extended to the operator T. The results are used to consider the existence of stationary optimal policies for Markov decision processes with multiplicative rewards.