Optimising crowdsourcing efficiency: Amplifying human computation with validation-Reference-Cited by-同舟云学术

Optimising crowdsourcing efficiency: Amplifying human computation with validation

Published:2018-02-01 Issue:1 Volume:60 Page:41-49
ISSN:2196-7032
Container-title:it - Information Technology
language:en
Short-container-title:

Author:

Chamberlain Jon¹,Kruschwitz Udo¹,Poesio Massimo²

Affiliation:

1. University of Essex , Wivenhoe Park , Colchester CO4 3SQ , UK

2. Queen Mary University , Mile End Rd , London E1 4NS , UK

Abstract

Abstract Crowdsourcing has revolutionised the way tasks can be completed but the process is frequently inefficient, costing practitioners time and money. This research investigates whether crowdsourcing can be optimised with a validation process, as measured by four criteria: quality; cost; noise; and speed. A validation model is described, simulated and tested on real data from an online crowdsourcing game to collect data about human language. Results show that by adding an agreement validation (or a like/upvote) step fewer annotations are required, noise and collection time are reduced and quality may be improved.

Funder

Engineering and Physical Sciences Research Council

Publisher

Walter de Gruyter GmbH

Subject

General Computer Science

Link

https://www.degruyter.com/document/doi/10.1515/itit-2017-0020/pdf

Reference19 articles.

1. R. Artstein and M. Poesio, Inter-coder agreement for computational linguistics, Computational Linguistics 34 (2008), 555–596.10.1162/coli.07-034-R2

2. Yochai Benkler and Helen Nissenbaum, Commons-based Peer Production and Virtue, Journal of Political Philosophy 14 (2006), 394–419.10.1111/j.1467-9760.2006.00235.x

3. Michael S. Bernstein, Greg Little, Robert C. Miller, Björn Hartmann, Mark S. Ackerman, David R. Karger, David Crowell and Katrina Panovich, Soylent: A Word Processor with a Crowd Inside, in: Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology (UIST’10), pp. 313–322, 2010.

4. Amiangshu Bosu, Christopher S. Corley, Dustin Heaton, Debarshi Chatterji, Jeffrey C. Carver and Nicholas A. Kraft, Building Reputation in StackOverflow: An Empirical Investigation, in: Proceedings of the 10th Working Conference on Mining Software Repositories (MSR’13), pp. 89–92, 2013.

5. Daren C. Brabham, Crowdsourcing, The MIT Press, 2013.

Cited by 5 articles. 订阅此论文施引文献订阅此论文施引文献，注册后可以免费订阅5篇论文的施引文献，订阅后可以查看论文全部施引文献

1. Human-centred design on crowdsourcing annotation towards improving active learning model performance;Journal of Information Science;2023-10-31

2. A survey of mobile crowdsensing and crowdsourcing strategies for smart mobile device users;CCF Transactions on Pervasive Computing and Interaction;2022-07-15

3. 2. Strategic Application of Clinical Practice Guideline for CKD Management;Nihon Naika Gakkai Zasshi;2020-09-10

4. A Review of Mobile Crowdsourcing Architectures and Challenges: Toward Crowd-Empowered Internet-of-Things;IEEE Access;2019

5. Using Social Media for Biomonitoring: How Facebook, Twitter, Flickr and Other Social Networking Platforms Can Provide Large-Scale Biodiversity Data;Advances in Ecological Research;2018