Large-scale volunteer computing over the Internet

Abstract

Abstract Cycle sharing over the Internet has increased in popularity during the last decade, with increasingly powerful machines being made available to existing projects. In this paper, we present GiGi-MR, a framework that allows non-expert users to run CPU-intensive jobs on top of volunteer resources over the Internet. GiGi-MR has several distinctive features: it allows non-expert users to easily partition their jobs in several parallel tasks; such Bag-of-Tasks (BoT) are executed in parallel as a set of MapReduce applications; the volunteer resources that provide the best match for the tasks being executed are chosen (using attenuated bloom filters); it provides a portable checkpointing fault-tolerance mechanism based on virtualization; it does not rely exclusively on a central server (or servers) at all times (thus minimizing the bottleneck effect); it deals with malicious participants (possibly byzantine) using an efficient partial replication mechanism to validate the results obtained; and it is compatible with BOINC (one of the most popular open-source software platforms for computing using volunteered resources). We describe GiGi-MR’s architecture and evaluate its performance by executing several MapReduce applications on a wide area testbed. Furthermore, we use micro-benchmarks to assess each one of GiGi-MR’s components independently. The system’s overhead is minimal. When compared to an unmodified volunteer computing system, GiGi-MR obtains a performance increase of over 60 % in application turnaround time, while reducing the bandwidth used by an order of magnitude.