Optimal Computing Budget Allocation for Data-Driven Ranking and Selection

Abstract

In a fixed-budget ranking and selection (R&S) problem, one aims to identify the best design among a finite number of candidates by efficiently allocating the given computing budget to evaluate design performance. Classical methods for R&S usually assume the distribution of the randomness in the system is exactly known. In this paper, we consider the practical scenario where the true distribution is unknown but can be estimated from streaming input data that arrive in batches over time. We formulate the R&S problem in this dynamic setting as a multistage problem where we adopt the Bayesian approach to estimate the distribution, and we formulate a stagewise optimization problem to allocate the computing budget. We characterize the optimality conditions for the stagewise problem by applying the large deviations theory to maximize the decay rate of the probability of false selection. Based on the optimality conditions and combined with the updating of distribution estimates, we design two sequential budget allocation procedures for R&S under streaming input data. We theoretically guarantee the consistency and asymptotic optimality of the proposed procedures. We demonstrate the practical efficiency through numerical experiments in comparison with the equal allocation policy and an extension of the optimal computing budget allocation algorithm. Funding: The authors gratefully acknowledge the support of the Air Force Office of Scientific Research [Grant FA9550-22-1-0244], the National Science Foundation [Grant NSF-DMS2053489], and the NSF AI Institute for Advances in Optimization under [Grant NSF-2112533]. Supplemental Material: The online appendix is available at https://doi.org/10.1287/ijoo.2024.0035 .