Abstract
AbstractThe lack of annotated publicly available medical images is a major barrier for innovations. At the same time, many de-identified images and much knowledge are shared by clinicians on public forums such as medical Twitter. Here we harness these crowd platforms to curate OpenPath, a large dataset of 208,414 pathology images paired with natural language descriptions. This is the largest public dataset for pathology images annotated with natural text. We demonstrate the value of this resource by developing PLIP, a multimodal AI with both image and text understanding, which is trained on OpenPath. PLIP achieves state-of-the-art zero-shot and transfer learning performances for classifying new pathology images across diverse tasks. Moreover, PLIP enables users to retrieve similar cases by either image or natural language search, greatly facilitating knowledge sharing. Our approach demonstrates that publicly shared medical information is a tremendous resource that can be harnessed to advance biomedical AI.
Publisher
Cold Spring Harbor Laboratory
Reference57 articles.
1. Huang Z , Shao W , Han Z , Alkashash AM , De la Sancha C , Parwani AV , et al. Artificial intelligence reveals features associated with breast cancer neoadjuvant chemotherapy responses from multi-stain histopathologic images. NPJ Precis Oncol. 2023;7: 14.
2. Werneck Krauss Silva V, Busam KJ;et al. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nat Med,2019
3. Dawood M , Branson K , Rajpoot NM , Ul Amir Afsar Minhas F. ALBRT: Cellular Composition Prediction in Routine Histology Images. 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW). IEEE; 2021. pp. 664–673.
4. Hegde N , Hipp JD , Liu Y , Emmert-Buck M , Reif E , Smilkov D , et al. Similar image search for histopathology: SMILY. NPJ Digit Med. 2019;2: 56.
5. Fast and scalable search of whole-slide images via self-supervised deep learning;Nat Biomed Eng,2022
Cited by
1 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献