Reflection on future directions: a systematic review of reported limitations and solutions in interactive information retrieval user studies

Abstract

PurposeThe purpose of this paper is to understand how users behave and evaluate how systems with users are essential for interactive information retrieval (IIR) research. User study methodology serves as a primary approach to answering IIR research questions. In addition to designing user study procedures, understanding the limitations of varying study designs and discussing solutions to the limitations is also critical for improving the methods and advancing the knowledge in IIR.Design/methodology/approachGiven this unresolved gap, we apply the faceted framework developed by Liu and Shah (2019) in systematically reviewing 131 IIR user studies recently published (2016–2021) on multiple IR and information science venues.FindingsOur study achieve three goals: (1) extracting and synthesizing the reported limitations on multiple aspects of user study (e.g. recruitment, tasks, study procedures, system interfaces, data analysis methods) under associated facets; (2) summarizing the reported solutions to the limitations; (3) clarifying the connections between types of limitations and types of solutions.Practical implicationsThe bibliography of user studies can be used by students and junior researchers who are new to user-centered IR studies as references for study design. Our results can facilitate the reflection and improvement on IR research methodology and serve as a checklist for evaluating customized IIR user studies in varying problem spaces.Originality/valueTo our knowledge, this work is the first study that systematically reviews the study limitations and solutions reported by IIR researchers in articles and empirically examines their connections to different study components.