Concordance assessment of a cognitive computing system in Thailand.

Abstract

6589 Background: IBM Watson for Oncology (WFO) was trained by Memorial Sloan Kettering and is a cognitive computing system that uses natural language processing to ingest patient data in structured and unstructured formats. The system provides physicians with treatment options that are derived from established guidelines, the medical literature, and training from patient cases. In this study, we assessed the degree of concordance between treatment recommendations proposed by WFO and oncologists at Bumrungrad International Hospital (BIH). BIH is a 580-bed multispecialty hospital in Bangkok, Thailand. Methods: Data from breast, colorectal, gastric, and lung cancer patients treated at BIH were entered into WFO in 2015 and 2016. Retrospective cases were entered after a treatment plan had been determined, and prospective cases were entered during patients’ treatment planning sessions. WFO recommendations were provided in 3 categories: “Recommended”, “For Consideration”, and “Not Recommended.” Concordance was analyzed by comparing the decisions made by the oncologists to those proposed by WFO. Concordance was achieved when the oncologist’s treatment suggestion was in the “Recommended” or “For Consideration” categories given by WFO. Results: A total of 211 cases were assessed, 92 were retrospective and 119 were prospective. The overall concordance rate was 83%; 89% for colorectal, 91% for lung, 76% for breast, and 78% for gastric cancer. Similar concordance rates were observed when retrospective and prospective cases were analyzed separately. Discordance was attributable in part to local oncologists’ preferences for non-U.S. guidelines for certain cancers, especially gastric cancer. Conclusions: There was a high degree of concordance between WFO treatment options and the decisions made by local oncologists. Similar results were recently reported in a breast cancer concordance study conducted using WFO in India (San Antonio Breast Cancer Symposium 2016, Somashekhar et al). WFO’s capabilities as a cognitive decision support tool can be further improved by incorporating regional guidelines. Future work will analyze reasons for discordance such as cost, insurance requirements, and patient and physician preference.