Assessment of the efficiency of a Chat GPT-based tool, MyGenAssist, in an industry pharmacovigilance department for case documentation: a cross-over study (Preprint)

Abstract

At the end of 2023, Bayer AG® launched its own internal large language model (LLM), MyGenAssist®, based on Chat GPT® technology to overcome data privacy concerns. It may offer the possibility to decrease their harshness and save time in repetitive and recurrent tasks that then could be dedicated to activities with higher added value. Although there is a current worldwide reflection whether Artificial Intelligence should be integrated to Pharmacovigilance, medical literature doesn’t provide enough data concerning LLMs and their daily applications in such a setting. Here, we studied how this tool could improve case documentation process, which is a duty for authorization holders as per European and French Good Vigilance Practices.

To test whether the use of a LLM could improve the Pharmacovigilance documentation process.

MyGenAssist® was trained to draft templates for case documentation letters meant to be sent to the reporters. Information provided within the template changes depending on the case: such data comes from a table sent to the LLM. We then measured the time spent on each case for a period of four months (2 months before using the tool and 2 months after its implementation). A multiple linear regression model was created with the time spent on each case as the explained variable, and all parameters that could influence this time were included as explanatory variables (use of MyGenAssist®, type of recipient, number of questions, user). To test if the use of this tool impacts the process, we compared the recipients’ response rate with and without the use of MyGenAssist®.

An average 23.3% (CI95: 13.8%-32.8%) time saving was made thanks to MyGenAssist® (P<.001, adjusted R-square=0.286) on each case, which could represent an average 10.7 working days saved each year. The answers’ rate wasn’t modified by the use of MyGenAssist® (41.67% vs 36.49%, P=.57), whether the recipient was a physician or a patient. Any significant difference was found regarding the time spent by the recipient to answer (2.20 vs 2.65 days after the last attempt of contact, P=.64). The implementation of MyGenAssist® for this activity only required a two-hour training of the PV Team.

Our study is the first to show that a Chat GPT-based tool can improve the efficiency of a GxP activity without needing a long training of the workforce. These first encouraging results could be an incentive for the implementation of LLM in other processes.