Survey on evaluation methods for dialogue systems-Reference-Cited by-同舟云学术

Survey on evaluation methods for dialogue systems

Published:2020-06-25 Issue:1 Volume:54 Page:755-810
ISSN:0269-2821
Container-title:Artificial Intelligence Review
language:en
Short-container-title:Artif Intell Rev

Author:

Deriu Jan^ORCID,Rodrigo Alvaro,Otegi Arantxa,Echegoyen Guillermo,Rosset Sophie,Agirre Eneko,Cieliebak Mark

Abstract

AbstractIn this paper, we survey the methods and concepts developed for the evaluation of dialogue systems. Evaluation, in and of itself, is a crucial part during the development process. Often, dialogue systems are evaluated by means of human evaluations and questionnaires. However, this tends to be very cost- and time-intensive. Thus, much work has been put into finding methods which allow a reduction in involvement of human labour. In this survey, we present the main concepts and methods. For this, we differentiate between the various classes of dialogue systems (task-oriented, conversational, and question-answering dialogue systems). We cover each class by introducing the main technologies developed for the dialogue systems and then present the evaluation methods regarding that class.

Funder

CHIST-ERA

Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung

Agencia Estatal de Investigación

Agence Nationale de la Recherche

Publisher

Springer Science and Business Media LLC

Subject

Artificial Intelligence,Linguistics and Language,Language and Linguistics

Link

http://link.springer.com/content/pdf/10.1007/s10462-020-09866-x.pdf

Reference202 articles.

1. Adiwardana D, Luong MT, So DR, Hall J, Fiedel N, Thoppilan R, Yang Z, Kulshreshtha A, Nemade G, Lu Y, et al. (2020) Towards a human-like open-domain chatbot. arXiv preprint arXiv:200109977

2. Ameixa D, Coheur L (2013) From subtitles to human interactions: introducing the SubTle Corpus. In: Technical report 2013

3. Austin JL (1962) How to do things with words. Oxford University Press, Oxford, William James

4. Banchs RE (2012) Movie-DiC: a Movie Dialogue Corpus for Research and Development. In: Proceedings of the 50th annual meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Association for Computational Linguistics, pp 203–207

5. Banchs RE, Li H (2012) IRIS: a chat-oriented dialogue system based on the vector space model. In: Proceedings of the ACL 2012 demonstrations, Jeju Island, Korea, pp 37–42

Cited by 113 articles. 订阅此论文施引文献订阅此论文施引文献，注册后可以免费订阅5篇论文的施引文献，订阅后可以查看论文全部施引文献

1. Dialogue agents 101: a beginner’s guide to critical ingredients for designing effective conversational systems;Natural Language Processing;2024-09-09

2. Leveraging large language models through natural language processing to provide interpretable machine learning predictions of mental deterioration in real time;Arabian Journal for Science and Engineering;2024-08-27

3. Behavior Alignment: A New Perspective of Evaluating LLM-based Conversational Recommendation Systems;Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval;2024-07-10

4. Doing Personal LAPS: LLM-Augmented Dialogue Construction for Personalized Multi-Session Conversational Search;Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval;2024-07-10

5. Rethinking the Evaluation of Dialogue Systems: Effects of User Feedback on Crowdworkers and LLMs;Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval;2024-07-10