On Item-Sampling Evaluation for Recommender System

Abstract

Personalized recommender system plays a crucial role in modern society, especially in e-commerce, news, and ads area. Correctly evaluating and comparing candidate recommendation models is as essential as constructing ones. The common offline evaluation strategy is holding out some user-interacted items from training data and evaluating the performance of recommendation models based on how many items they can retrieve. Specifically, for any hold-out item or so-called target item for a user, the recommendation models try to predict the probability that the user would interact with the item, and rank it among overall items , this is called global evaluation . Intuitively, a good recommendation model would assign high probabilities to such hold-out/target items. Based on the specific ranks, some metrics like Recall @ K and NDCG @ K can be calculated to further quantify the quality of the recommender model. Instead of ranking the target items among all items, Koren [22] first proposed to rank them among a small sampled set of items , and then quantified the performance of the models, this is called sampling evaluation . Ever since then, there has been a large amount of work adopting sampling evaluation due to its efficiency and frugality. In recent work, Rendle and Krichene [24, 32] argued that the sampling evaluation is ”inconsistent” with respect to a global evaluation in terms of offline top-k metrics. In this work, we first investigate the ”inconsistent” phenomenon by taking a glance at the connections between sampling evaluation and global evaluation. We reveal the approximately linear relationship between sampling with respect to its global counterpart in terms of the top-K Recall metric. Second, we propose a new statistical perspective of the sampling evaluation - to estimate the global rank distribution of the entire population. After the estimated rank distribution is obtained, the approximation of the global metric can be further derived. Third, we extend the work of Krichene and Rendle [24], directly optimizing the error with ground truth, providing not only a comprehensive empirical study but also a rigorous theoretical understanding of the proposed metric estimators. To address the ”blind spot” issue, where accurately estimating metrics for small top-K values in sampling evaluation is challenging, we propose a novel adaptive sampling method that generalizes the expectation-maximization (EM) algorithm to this setting. Last but not least, we also study the user sampling evaluation effect. This series of works outlines a clear roadmap for sampling evaluation and establishes a foundational theoretical framework. Extensive empirical studies validate the reliability of the sampling methods presented.