Collaborative Crowdsourced Software Testing

Abstract

Crowdsourced software testing (CST) uses a crowd of testers to conduct software testing. Currently, the microtasking model is used in CST; in it, a testing task is sent to individual testers who work separately from each other. Several studies mentioned that the quality of test reports produced by individuals was a drawback because a large number of invalid defects were submitted. Additionally, individual workers tended to catch the simple defects, not those with high complexity. This research explored the effect of having pairs of collaborating testers working together to produce one final test report. We conducted an experiment with 75 workers to measure the effect of this approach in terms of (1) the total number of unique valid defects detected, (2) the total number of invalid defects reported, and (3) the possibility of detecting more difficult defects. The findings show that testers who worked in collaborating pairs can be as effective in detecting defects as an individual worker; the differences between them are marginal. However, CST significantly affects the quality of test reports submitted in two dimensions: it helps reduce the number of invalid defects and also helps detect more difficult defects. The findings are promising and suggest that CST platforms can benefit from new mechanisms that allow for the formation of teams of two individuals who can participate in doing testing jobs.