Job Recommendations: Benchmarking of Collaborative Filtering Methods for Classifieds

Abstract

Classifieds pose numerous challenges for recommendation methods, including the temporary visibility of ads, the anonymity of most users, and the fact that typically only one user can consume an advertised item. In this work, we address these challenges by choosing models and evaluation procedures that are considered accurate, diverse, and efficient (in terms of memory and time consumption during training and prediction). This paper aims to benchmark various recommendation methods for job classifieds, using OLX Jobs as an example, to enhance the conversion rate of advertisements and user satisfaction. In our research, we implement scalable methods and represent different approaches to the recommendations: Alternating Least Square (ALS), LightFM, Prod2Vec, RP3Beta, and Sparse Linear Methods (SLIM). We conducted A/B tests by sending millions of messages with recommendations to perform online evaluations of selected methods. In addition, we have published the dataset created for our research. To the best of our knowledge, this is the first dataset of its kind. It contains 65,502,201 events performed on OLX Jobs by 3,295,942 users who interacted with (displayed, replied to, or bookmarked) 185,395 job ads over two weeks in 2020. We demonstrate that RP3Beta, SLIM, and ALS perform significantly better than Prod2Vec and LightFM when tested in a laboratory setting. Online A/B tests also show that sending messages with recommendations generated by the ALS and RP3Beta models increases the number of users contacting advertisers. Additionally, RP3Beta had a 20% more significant impact on this metric than ALS.