On the number of trials needed to distinguish similar alternatives-Reference-Cited by-同舟云学术

On the number of trials needed to distinguish similar alternatives

Published:2022-07-28 Issue:31 Volume:119 Page:
ISSN:0027-8424
Container-title:Proceedings of the National Academy of Sciences
language:en
Short-container-title:Proc. Natl. Acad. Sci. U.S.A.

Author:

Chierichetti Flavio¹^ORCID,Kumar Ravi²^ORCID,Tomkins Andrew²^ORCID

Affiliation:

1. Dipartimento di Informatica, Sapienza University of Rome, Rome, Italy 00185;

2. Google, Mountain View, CA 94043

Abstract

A/B testing is widely used to tune search and recommendation algorithms, to compare product variants as efficiently and effectively as possible, and even to study animal behavior. With ongoing investment, due to diminishing returns, the items produced by the new alternative B show smaller and smaller improvement in quality from the items produced by the current system A. By formalizing this observation, we develop closed-form analytical expressions for the sample efficiency of a number of widely used families of slate-based comparison tests. In empirical trials, these theoretical sample complexity results are shown to be predictive of real-world testing efficiency outcomes. These findings offer opportunities for both more cost-effective testing and a better analytical understanding of the problem.

Publisher

Proceedings of the National Academy of Sciences

Subject

Multidisciplinary

Link

https://pnas.org/doi/pdf/10.1073/pnas.2202116119

Reference20 articles.

1. Efficient search for a face by chimpanzees (Pan troglodytes)

2. A law of comparative judgment.

3. D. Ennis, Thurstonian Models–Categorical Decision Making in the Presence of Noise (The Institute for Perception, Richmond, VA, 2016).

4. The perception of two-tone Mooney faces in chimpanzees (Pan troglodytes)

5. J. H. Bak, J. Y. Choi, A. Akrami, I. Witten, J. W. Pillow, “Adaptive optimal training of animal behavior” Advances in Neural Information Processing Systems 29, D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, R. Garnett, Eds. (NIPS, 2016), pp. 1947–1955.