Report on the 8th Workshop on Search-Oriented Conversational Artificial Intelligence (SCAI 2024) at CHIIR 2024-Reference-Cited by-同舟云学术

Report on the 8th Workshop on Search-Oriented Conversational Artificial Intelligence (SCAI 2024) at CHIIR 2024

Published:2024-06 Issue:1 Volume:58 Page:1-12
ISSN:0163-5840
Container-title:ACM SIGIR Forum
language:en
Short-container-title:SIGIR Forum

Author:

Frummet Alexander¹,Papenmeier Andrea²,Fröbe Maik³,Kiesel Johannes⁴,Adlakha Vaibhav⁵,Braunschweiler Norbert⁶,Dubiel Mateusz⁷,Ghosh Satanu⁸,Gohsen Marcel⁴,Kreutz Christin⁹,Momeni Milad¹⁰,Nilles Markus¹¹,Cherumanal Sachin Pathiyan¹²,Pirmoradi Abbas¹⁰,Thomas Paul¹³,Trippas Johanne R.¹²,Zelch Ines¹⁴,Zendel Oleg¹²

Affiliation:

1. Universität Regensburg, Germany

2. University of Twente, Netherlands

3. Friedrich-Schiller-Universität Jena, Germany

4. Bauhaus-Universität Weimar, Germany

5. Mila-Quebec AI Institute, Canada

6. Toshiba Europe Limited, UK

7. University of Luxembourg, Luxembourg

8. University of New Hampshire, US

9. TH Mittelhessen, Germany

10. University of Regina, Canada

11. Trier University, Germany

12. RMIT University, Australia

13. Microsoft, Australia

14. Leipzig University, Germany

Abstract

Conversational Agents are increasingly integrated into our daily routines, assisting us with various tasks, from simple commands such as scheduling events to more complex conversational search interactions. Such conversational search systems are traditionally evaluated with word-overlap metrics such as F1 score and accuracy. The full-day workshop on Search-Oriented Conversational Artificial Intelligence (SCAI) at CHIIR 2024 explored the evaluation of conversational search systems from the user's perspective. This interactive workshop included multiple panel discussions and working groups focused on developing and discussing innovative, user-centered evaluation methods for these systems. This paper, co-authored by both organizers and participants of the workshop, presents a summary of the insights gathered from the panel discussions and working groups. Date : 14 March 2024. Website : https://scai.info/scai-2024/.

Publisher

Association for Computing Machinery (ACM)

Link

https://dl.acm.org/doi/pdf/10.1145/3687273.3687282

Reference17 articles.

1. Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI

2. Prosody-based automatic detection of annoyance and frustration in human-computer dialog

3. User Simulation for Evaluating Information Access Systems

4. Satanjeev Banerjee and Alon Lavie. METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. In Jade Goldstein, Alon Lavie, Chin-Yew Lin, and Clare R. Voss, editors, Proceedings of the Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization@ACL 2005, Ann Arbor, Michigan, USA, June 29, 2005, pages 65--72. Association for Computational Linguistics, 2005. URL https://aclanthology.org/W05-0909/.

5. Search-Oriented Conversational AI (SCAI)