Online Sequencing of Non-Decomposable Macrotasks in Expert Crowdsourcing

Abstract

We introduce the problem of Task Assignment and Sequencing, which models online optimization in expert crowdsourcing settings that involve non-decomposable macrotasks. Non-decomposition is a property of certain types of complex problems, like the formulation of an R&D approach or the definition of a research methodology, which cannot be handled through the "divide-and-conquer" approach typically used in microtask crowdsourcing. In contrast to splitting the macrotask to multiple microtasks and allocating them to several workers in parallel, our model supports the sequential improvement of the macrotask one worker at a time, across distinct time slots of a given timeline, until a sufficient quality level is achieved. Our model assumes an online environment where expert workers are available only at specific time slots and worker/task arrivals are not known a priori . With respect to this setting, we propose TAS-ONLINE, an online algorithm that aims to complete as many tasks as possible within budget, required quality, and a given timeline, without any future input information regarding job release dates or worker availabilities. Experimental results comparing TAS-ONLINE to five benchmarks show that it achieves more completed jobs, lower flow times, and higher job quality. This work bears practical implications for providing performance and quality guarantees to expert crowdsourcing platforms that wish to integrate non-decomposable macrotasks into their offered services.