Measuring Annotator Agreement Generally across Complex Structured, Multi-object, and Free-text Annotation Tasks-Reference-Cited by-同舟云学术

Measuring Annotator Agreement Generally across Complex Structured, Multi-object, and Free-text Annotation Tasks

Published:2022-04-25 Issue: Volume: Page:
ISSN:
Container-title:Proceedings of the ACM Web Conference 2022
language:
Short-container-title:

Author:

Braylan Alexander¹,Alonso Omar²,Lease Matthew¹

Affiliation:

1. The University of Texas at Austin, USA

2. Northeastern University, USA

Funder

Micron Foundation

Publisher

ACM

Link

https://dl.acm.org/doi/pdf/10.1145/3485447.3512242

Reference48 articles.

1. Omar Alonso . 2013. Implementing crowdsourcing-based relevance experimentation: an industrial perspective. Information retrieval 16, 2 ( 2013 ), 101–120. Omar Alonso. 2013. Implementing crowdsourcing-based relevance experimentation: an industrial perspective. Information retrieval 16, 2 (2013), 101–120.

2. The Practice of Crowdsourcing

3. Weighted Krippendorff's alpha is a more reliable metrics for multi-coders ordinal annotations: experimental studies on emotion, opinion and coreference annotation

4. OpenCrowd: A Human-AI Collaborative Approach for Finding Social Influencers via Open-Ended Answers Aggregation

5. Data excellence for AI

Cited by 10 articles. 订阅此论文施引文献订阅此论文施引文献，注册后可以免费订阅5篇论文的施引文献，订阅后可以查看论文全部施引文献

1. SPRAG: building and benchmarking a Short Programming-Related Answer Grading dataset;International Journal of Data Science and Analytics;2024-06-04

2. Data Sorting Influence on Short Text Manual Labeling Quality for Hierarchical Classification;Big Data and Cognitive Computing;2024-04-07

3. An entity-centric approach to manage court judgments based on Natural Language Processing;Computer Law & Security Review;2024-04

4. Some Useful Things to Know When Combining IR and NLP: The Easy, the Hard and the Ugly;Proceedings of the 17th ACM International Conference on Web Search and Data Mining;2024-03-04

5. Toward a generalizable machine learning workflow for neurodegenerative disease staging with focus on neurofibrillary tangles;Acta Neuropathologica Communications;2023-12-18