Affiliation:
1. Renmin University of China, Beijing, China
2. Beijing University of Posts and Telecommunications, Beijing, China
Abstract
In recommender systems, top-
N
recommendation is an important task with implicit feedback data. Although the recent success of deep learning largely pushes forward the research on top-
N
recommendation, there are increasing concerns on appropriate evaluation of recommendation algorithms. It therefore is important to study how recommendation algorithms can be reliably evaluated and thoroughly verified. This work presents a large-scale, systematic study on six important factors from three aspects for evaluating recommender systems. We carefully select 12 top-
N
recommendation algorithms and eight recommendation datasets. Our experiments are carefully designed and extensively conducted with these algorithms and datasets. In particular, all the experiments in our work are implemented based on an open sourced recommendation library, Recbole [
139
], which ensures the reproducibility and reliability of our results. Based on the large-scale experiments and detailed analysis, we derive several key findings on the experimental settings for evaluating recommender systems. Our findings show that some settings can lead to substantial or significant differences in performance ranking of the compared algorithms. In response to recent evaluation concerns, we also provide several suggested settings that are specially important for performance comparison.
Funder
National Natural Science Foundation of China
Beijing Natural Science Foundation
Beijing Outstanding Young Scientist Program
Beijing Academy of Artificial Intelligence
Publisher
Association for Computing Machinery (ACM)
Subject
Computer Science Applications,General Business, Management and Accounting,Information Systems
Cited by
6 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献