Trainee versus supervisor viewpoints of entrustment: using artificial intelligence language models to detect thematic differences and potential biases

Abstract

Abstract The entrustment framework redirects assessment from considering only trainees’ competence to decision-making about their readiness to perform clinical tasks independently. Since trainees’ and supervisors’ viewpoints both contribute to entrustment decisions, we examined how they may differentially prioritize factors that determine trust, and how biases may influence this process. Under a social constructivist paradigm, we developed large language model (LLM) based approach to examine feedback dialogs (N = 24187, each with an associated entrustment rating) between student trainees and their precepting supervisors. Using LLM-assisted factor analysis, we compared how trainees and supervisors documented similar types of clinical tasks by identifying factors correlated with entrustment ratings. Supervisors’ factors were dominated by skills related to patient presentations, while trainees’ factors captured a wider range of themes, including both clinical performance and personal qualities. To examine bias, we developed a gender-neutral LLM to measure sentiment in feedback narratives. On average, trainees used more negative language (5.3% lower probability of positive sentiment, p < 0.05) compared to supervisors, while assigning themselves a higher entrustment rating (+ 0.08 on a 1–4 scale, p < 0.05). Trainees’ documentation reflected more positive sentiment in the case of male trainees (+ 1.3%, p < 0.05) and of trainees underrepresented in medicine (+ 1.3%, p < 0.05). Importantly, entrustment ratings themselves did not appear to reflect these biases, neither when documented by the trainee nor supervisor. As such, bias appeared to affect trainee self-perceptions more than the degree of entrustment they experienced. Mitigating these biases is nonetheless important because they may affect trainees’ assimilation into their roles and formation of trusting relationships.