An efficient system using implicit feedback and lifelong learning approach to improve recommendation

Abstract

AbstractThis paper presents a new technique for contextual item-to-item Collaborative Filtering-based Recommender System, an improved version popularised by e-commerce giantAmazontwo decades back. The concept is based on items also-viewed under the same browsing session. Users’ browsing patterns, locations, and timestamps are considered as thecontextandlatent factorsfor each user. The algorithm computes recommendations based on users’implicitendorsements byclicks. The algorithm does not enforce the user to log in to provide recommendations and is capable of providing accurate recommendations fornon-logged-in usersand with a setting where the system is unaware of users’ preferences and profile data (non-logged-in users). This research takes the cue from humanlifelong incremental learningexperience applied to machine learning on a large volume of the data pool. First, all historical data is gathered from collectable sources in a distributed manner through big data tools. Then, a long-running batch job creates the initial model and saves it to Hadoop Distributed File System (HDFS). An ever-running streaming job loads the model from HDFS and builds on top of it in anincrementalfashion. At the architectural level, this resembles the big data mix processingLambda Architecture. The recommendation is computed based on a proposed equation for a weighted sum between near real-time and historical batch data. Real-time and batch processing engines act as autonomous Multi-agent systems in collaboration. We propose an ensemble method for batch-stream the recommendation engine. We introduce a novelLifelong Learning Modelfor recommendation through Multi-agent Lambda Architecture. The recommender system incrementally updates its model on streaming datasets to improve over time.