Using Natural Language and Health Ontologies in Hope Recommender System: Evaluation of Use in Primary Care-Reference-Cited by-同舟云学术

Using Natural Language and Health Ontologies in Hope Recommender System: Evaluation of Use in Primary Care

Published:2024-04-09 Issue: Volume: Page:
ISSN:
Container-title:
language:
Short-container-title:

Author:

Eguia Hans¹,Sánchez-Bocanegra Carlos¹,Fernandez-Llatas Carlos²,Alvarez-López Fernando³,Saigí-Rubió Francesc¹

Affiliation:

1. Universitat Oberta de Catalunya (UOC)

2. SABIEN Group at ITACA institute at Universitat Politècnica de València

3. Universidad de Manizales (UM)

Abstract

Abstract Objective To evaluate the accuracy of the results of an evidence-based information recommender called Health Operation for Personalized Evidence (HOPE), which is a UMLS ontology-based solution and was used as an automatic assistant to find information in primary care cases. Materials and methods Ontologies and Natural Language Processing (NLP) were used by HOPE to assess clinical cases and generate a group of recommendations. The results were reviewed by fifty general practitioners (raters). The Kappa-Fleiss Coefficient measured the level of agreement among them. The precision was also measured to evaluate the satisfaction of a user who is presented with different EBM references. The precision (precision@k) and Normalized Discounted Cumulative Gain (NDCG@k) were also calculated because both metrics help assess different aspects of AI recommendation systems. Results The Kappa Fleiss coefficient for the 50 raters was 0,66 (z=277 and p-value=0) which represents an “important association” among raters. The results for precision@3, precision@4 and precision@5 over 0,75 showed also good results (@3 72%; @4 72%; @5 68%) as well as the NDCG@k (NDCG@3 63%; NDCG@4 60%; NDCG@5 60%). The results presented after the NDCG@k indicate that the recommendation system adequately classifies the recommendations presented by HOPE according to the ratings of the raters. The results are highly relevant and they are in the optimal order according to the preferences of raters. HOPE’s response service time seems to be almost immediate, with a mean time of 17,4 seconds. Conclusion The recommender system seems to find accurate information for primary care. The use of UMLS ontologies and NLP allows HOPE to find useful information for a clinical case within a reasonable time (relative to the patient’s time assigned). These recommendations could potentially get better diagnosis and treatment as well as eventually reduce consultation time.

Publisher

Research Square Platform LLC

Reference25 articles.

1. Atallah ÁN. Evidence-based medicine. Sao Paulo Med J [Internet]. 2018 Mar 1 [cited 2023 Apr 8];136(2):99–100. Available from: http://www.scielo.br/j/spmj/a/3gRb38NTGTcgnbqYjW8bzcc/?lang=en

2. Djulbegovic B, Guyatt GH. Progress in evidence-based medicine: a quarter century on. Lancet [Internet]. 2017 Jul 22 [cited 2023 Apr 8];390(10092):415–23. Available from: http://www.thelancet.com/article/S0140673616315926/fulltext

3. Lin C, Hsu CJ, Lou YS, Yeh SJ, Lee CC, Su SL, et al. Artificial Intelligence Learning Semantics via External Resources for Classifying Diagnosis Codes in Discharge Notes. J Med Internet Res [Internet]. 2017 Nov 1 [cited 2023 Apr 8];19(11). Available from: /pmc/articles/PMC5696581/

4. Olex AL, McInnes BT. Review of Temporal Reasoning in the Clinical Domain for Timeline Extraction: Where we are and where we need to be. J Biomed Inform [Internet]. 2021;118:103784. Available from: https://linkinghub.elsevier.com/retrieve/pii/S1532046421001131

5. Suh HS, Tully JL, Meineke MN, Waterman RS, Gabriel RA. Identification of Preanesthetic History Elements by a Natural Language Processing Engine. Anesth Analg [Internet]. 2022 Dec 1 [cited 2023 Apr 2];135(6):1162–71. Available from: https://journals.lww.com/anesthesia-analgesia/Fulltext/2022/12000/Identification_of_Preanesthetic_History_Elements.10.aspx