Author:
Blumenthal Oded,Shani Guy
Abstract
AbstractPartially observable Markov decision processes (POMDP) are a useful model for decision-making under partial observability and stochastic actions. Partially Observable Monte-Carlo Planning (POMCP) is an online algorithm for deciding on the next action to perform, using a Monte-Carlo tree search approach, based on the UCT algorithm for fully observable Markov-decision processes. POMCP develops an action-observation tree, and at the leaves, uses a rollout policy to provide a value estimate for the leaf. As such, POMCP is highly dependent on the rollout policy to compute good estimates, and hence identify good actions. Thus, many practitioners who use POMCP are required to create strong, domain-specific heuristics. In this paper, we model POMDPs as stochastic contingent planning problems. This allows us to leverage domain-independent heuristics that were developed in the planning community. We suggest two heuristics, the first is based on the well-known $$h_{add}$$
h
add
heuristic from classical planning, and the second is computed in belief space, taking the value of information into account.
Funder
Israel Science Foundation
Ben-Gurion University
Publisher
Springer Science and Business Media LLC
Reference63 articles.
1. Agrawal, Rajeev: Sample mean based index policies by o (log n) regret for the multi-armed bandit problem. Adv. Appl. Probab. 27(4), 1054–1078 (1995)
2. Alagoz, O., Hsu, H., Schaefer, A.J., Roberts, M.S.: Markov decision processes: a tool for sequential decision making under uncertainty. Med. Decision Making 30(4), 474–483 (2010)
3. Albore, A., Geffner, H.: Acting in partially observable environments when achievement of the goal cannot be guaranteed. In: Workshop on Planning and Plan Execution for Real-World Systems, at 19th Int. Conf. on Planning and Scheduling (ICAPS ’09). (2009)
4. Albore, A., Palacios, H., Geffner, H.: A translation-based approach to contingent planning. In: IJCAI 2009, Proceedings of the 21st International Joint Conference on Artificial Intelligence, Pasadena, California, USA, July 11-17, 2009. pp. 1623–1628 (2009)
5. Albore, A., Palacios, H., Geffner, H.: A translation-based approach to contingent planning. In: IJCAI, vol. 9. pp. 1623–1628 (2009). https://doi.org/10.5555/1661445.1661706, https://dl.acm.org/doi/10.5555/1661445.1661706