Observe Before Play

Abstract

We consider the stochastic multi-armed bandit (MAB) problem in a setting where a player can, at a cost, pre-observe one or multiple arms before playing one of them in each round. Apart from the classic trade-off between exploration (trying out more arms to find the best one) and exploitation (sticking with the arm believed to offer the highest reward), we encounter an additional dilemma in each single round, i.e., pre-observing more arms gives a higher chance to play the best one, but incurs a larger cost which decreases the overall reward. We design an Observe-Before-Play (OBP) policy for arms with Bernoulli rewards, which could be generalized to any i.i.d. reward distributions bounded in [0, 1]. Our strategy could enable a better policy for secondary spectrum access in Cognitive Ratio Networks, where users can sense multiple channels' occupancies before choosing one on which to transmit. To evaluate our policy, we define the regret as the gap between the expected overall reward gained by our OBP policy and that obtained by the expected optimum, which always chooses an optimal sequence of arms to pre-observe based on the perfect knowledge of the arm distributions. Experiments show that our OBP policy has sub-linear regret and can outperform the classical MAB algorithm when the cost of pre-observations is relatively low.