Author:
Buschmeyer Katharina,Hatfield Sarah,Zenner Julie
Abstract
This study aimed to develop an evaluation tool that assesses the use of AI-based decision support systems (DSSs) in professional practice from a human-centered perspective. Following the International Organization for Standardization, this perspective aims to ensure that the use of interactive technologies improves users' psychological load experience and behavior, e.g., in the form of reduced stress experience or increased performance. Concomitantly, this perspective attempts to proactively prevent or detect and correct the potential negative effects of these technologies on user load, such as impaired satisfaction and engagement, as early as possible. Based on this perspective, we developed and validated a questionnaire instrument, the Psychological Assessment of AI-based DSSs (PAAI), for the user-centered evaluation of the use of AI-based DSSs in practice. In particular, the instrument considers central design characteristics of AI-based DSSs and the corresponding work situation, which have a significant impact on users' psychological load. The instrument was tested in two independent studies. In Study 1, N = 223 individuals were recruited. Based on the results of item and scale analyses and an exploratory factor analysis, the newly developed instrument was refined, and the final version was tested using a confirmatory factor analysis. Findings showed acceptable-to-good fit indices, confirming the factorial validity of the PAAI. This was confirmed in a second study, which had N = 471 participants. Again, the CFA yielded acceptable-to-good fit indices. The validity was further confirmed using convergent and criterion validity analyses.
Funder
Bundesministerium für Arbeit und Soziales
Reference150 articles.
1. Understanding key drivers affecting students' use of artificial intelligence-based voice assistants;Al Shamsi;Educ. Inf. Technol,2022
2. “Critical success factors for implementing artificial intelligence (AI) projects in Dubai Government United Arab Emirates (UAE) health sector: Applying the extended technology acceptance model (TAM),”;Alhashmi;Proceedings of the International Conference on Advanced Intelligent Systems and Informatics 2019,2020
3. “Examining the main mobile learning system drivers' effects: a Mix empirical examination of both the expectation-confirmation model (ECM) and the technology acceptance model (TAM),”;Alshurideh;Proceedings of the International Conference on Advanced Intelligent Systems and Informatics 2019,2020
4. “How can you verify that I am using AI? Complementary frameworks for describing and evaluating AI-based digital agents in their usage contexts,”;Alter;56th Hawaii Conference on System Sciences (HICSS),2023
5. User satisfaction with mobile websites: the impact of perceived usefulness (PU), perceived ease of use (PEOU) and trust;Amin;Nankai Bus. Rev. Int,2014