An architecture and platform for developing distributed recommendation algorithms on large-scale social networks

Abstract

The creation of new and better recommendation algorithms for social networks is currently receiving much attention owing to the increasing need for new tools to assist users. The volume of available social data as well as experimental datasets force recommendation algorithms to scale to many computers. Given that social networks can be modelled as graphs, a distributed graph-oriented support able to exploit computer clusters arises as a necessity. In this work, we propose an architecture, called Lightweight-Massive Graph Processing Architecture, which simplifies the design of graph-based recommendation algorithms on clusters of computers, and a Java implementation for this architecture composed of two parts: Graphly, an API offering operations to access graphs; and jLiME, a framework that supports the distribution of algorithm code and graph data. The motivation behind the creation of this architecture is to allow users to define recommendation algorithms through the API and then customize their execution using job distribution strategies, without modifying the original algorithm. Thus, algorithms can be programmed and evaluated without the burden of thinking about distribution and parallel concerns, while still supporting environment-level tuning of the distributed execution. To validate the proposal, the current implementation of the architecture was tested using a followee recommendation algorithm for Twitter as case study. These experiments illustrate the graph API, quantitatively evaluate different job distribution strategies w.r.t. recommendation time and resource usage, and demonstrate the importance of providing non-invasive tuning for recommendation algorithms.