Affiliation:
1. Division of Otolaryngology–Head and Neck Surgery University of Alberta Edmonton Alberta Canada
2. Copula AI New York New York USA
3. Faculty of Medicine University of Alberta Edmonton Alberta Canada
4. Alberta Machine Intelligence Institute Edmonton Alberta Canada
5. Reeder AI New York New York USA
Abstract
AbstractObjectiveThe recent surge in popularity of large language models (LLMs), such as ChatGPT, has showcased their proficiency in medical examinations and potential applications in health care. However, LLMs possess inherent limitations, including inconsistent accuracy, specific prompting requirements, and the risk of generating harmful hallucinations. A domain‐specific model might address these limitations effectively.Study DesignDevelopmental design.SettingVirtual.MethodsOtolaryngology–head and neck surgery (OHNS) relevant data were systematically gathered from open‐access Internet sources and indexed into a knowledge database. We leveraged Retrieval‐Augmented Language Modeling to recall this information and utilized it for pretraining, which was then integrated into ChatGPT4.0, creating an OHNS‐specific knowledge question & answer platform known as ChatENT. The model is further tested on different types of questions.ResultsChatENT showed enhanced performance in the analysis and interpretation of OHNS information, outperforming ChatGPT4.0 in both the Canadian Royal College OHNS sample examination questions challenge and the US board practice questions challenge, with a 58.4% and 26.0% error reduction, respectively. ChatENT generated fewer hallucinations and demonstrated greater consistency.ConclusionTo the best of our knowledge, ChatENT is the first specialty‐specific knowledge retrieval artificial intelligence in the medical field that utilizes the latest LLM. It appears to have considerable promise in areas such as medical education, patient education, and clinical decision support. The model has demonstrated the capacity to overcome the limitations of existing LLMs, thereby signaling a future of more precise, safe, and user‐friendly applications in the realm of OHNS and other medical fields.
Reference35 articles.
1. Understanding the capabilities, limitations, and societal impact of large language models;Tamkin A;arXiv,2021
2. Evaluating Large language models trained on code;Chen M;arXiv,2021
3. Emergent abilities of large language models;Wei J;arXiv,2022
4. Are large language models ready for healthcare? a comparative study on clinical language understanding;Wang Y;arXiv,2023
5. The practical implementation of artificial intelligence technologies in medicine
Cited by
1 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献