Crowdsourcing for relevance evaluation-Reference-Cited by-同舟云学术

Crowdsourcing for relevance evaluation

Published:2008-11-30 Issue:2 Volume:42 Page:9-15
ISSN:0163-5840
Container-title:ACM SIGIR Forum
language:en
Short-container-title:SIGIR Forum

Author:

Alonso Omar¹,Rose Daniel E.¹,Stewart Benjamin¹

Affiliation:

1. A9.com, Palo Alto, CA

Abstract

Relevance evaluation is an essential part of the development and maintenance of information retrieval systems. Yet traditional evaluation approaches have several limitations; in particular, conducting new editorial evaluations of a search system can be very expensive. We describe a new approach to evaluation called TERC, based on the crowdsourcing paradigm, in which many online users, drawn from a large community, each performs a small evaluation task.

Publisher

Association for Computing Machinery (ACM)

Subject

Hardware and Architecture,Management Information Systems

Link

https://dl.acm.org/doi/pdf/10.1145/1480506.1480508

Reference9 articles.

1. Amazon Mechanical Turk http://www.mturk.com Amazon Mechanical Turk http://www.mturk.com

2. AI Gets a Brain

3. Brendan O'Connor "Search Engine Relevance: An Empirical Test" http://blog.doloreslabs.com/2008/04/search-engine-relevance-an-empirical-test/#more-35 accessed April 13 2008. Brendan O'Connor "Search Engine Relevance: An Empirical Test" http://blog.doloreslabs.com/2008/04/search-engine-relevance-an-empirical-test/#more-35 accessed April 13 2008.

4. Jeff Howe. "The Rise of Crowdsourcing". Wired June 2006. http://www.wired.com/wired/archive/14.06/crowds.html Jeff Howe. "The Rise of Crowdsourcing". Wired June 2006. http://www.wired.com/wired/archive/14.06/crowds.html

Cited by 165 articles. 订阅此论文施引文献订阅此论文施引文献，注册后可以免费订阅5篇论文的施引文献，订阅后可以查看论文全部施引文献

1. Rethinking the Evaluation of Dialogue Systems: Effects of User Feedback on Crowdworkers and LLMs;Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval;2024-07-10

2. Crowdsourcing: An Education Framework;International Journal of Innovative Science and Research Technology (IJISRT);2024-04-08

3. On the Impact of Showing Evidence from Peers in Crowdsourced Truthfulness Assessments;ACM Transactions on Information Systems;2024-01-22

4. A Preference Judgment Tool for Authoritative Assessment;Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval;2023-07-18

5. On the role of human and machine metadata in relevance judgment tasks;Information Processing & Management;2023-03