Utilizing Large Language Models for Named Entity Recognition in Traditional Chinese Medicine against COVID-19 Literature: Comparative Study (Preprint)

Abstract

Recent advances in large language models (LLMs) have shown remarkable performance on various downstream tasks in zero- and few-shot scenarios, shedding light on named entity recognition (NER) in low-resource domains. Traditional Chinese medicine (TCM) against COVID-19 has been a new research topic and led to niche research literature. NER techniques are crucial for extracting and utilizing the rich knowledge in such literature.

To explore and compare the performance of ChatGPT and other state-of-the-art LLMs on domain-specific NER tasks covering different entity types and domains in TCM against COVID-19 literature.

We established a dataset of 389 articles on TCM against COVID-19, and manually annotated 48 of them with 6 types of entities belonging to 3 domains as the ground truth, against which the NER performance of LLMs can be assessed. We then performed NER tasks for the 6 entity types using ChatGPT (GPT-3.5 and GPT-4) and 4 state-of-the-art BERT-based question-answering (QA) models (RoBERTa, MiniLM, PubMedBERT and SciBERT) without prior training on the specific task. A domain fine-tuned model (GSAP-NER) was also applied for a comprehensive comparison on one of the entity types. In the task setup, we considered two different matching methods, namely exact match and fuzzy match, to have better coverage of potential application scenarios. The evaluation metrics we used are Precision, Recall, and F-1 in both exact match and fuzzy match.

The overall performance of LLMs varied significantly in exact match and fuzzy match. In the fuzzy match, ChatGPT surpassed BERT-based QA models in 5 out of 6 tasks, while in exact match, BERT-based QA models outperformed ChatGPT in 5 out of 6 tasks but with a smaller F-1 difference. GPT-4 showed a significant advantage over other models in fuzzy match, especially on the entity type of TCM formula and the Chinese patent drug (TFD) and ingredient (IG), achieving a higher F-1 score of 0.814 and 0.689, respectively. Although GPT-4 outperformed BERT-based models on entity type of herb (HB, 0.324), target (TG, 0.319), and research method (RM, 0.443), none of the F-1 scores exceeded 0.5. GSAP-NER, outperformed GPT-4 in terms of F-1 by a slight margin (0.45 vs 0.433) on RM. ChatGPT achieved considerably higher recalls than precisions, particularly in the fuzzy match.

The NER performance of LLMs is highly dependent on the entity type, and their performance varies across application scenarios. ChatGPT could be a good choice for scenarios where high recall is favored, e.g., for domain novices obtaining an extensive overview of the field. However, for knowledge acquisition in rigorous scenarios, neither ChatGPT nor BERT-based QA models are off-the-shelf tools for professional practitioners. Besides, these BERT-based models are open-source, and thus more accessible for scientific inquiry and hence worth further exploration.