Testing generalizability of Crystal Bone, an AI/ML osteoporotic fracture risk prediction algorithm, using real-world evidence from three US health systems (Preprint)

Abstract

Real-world osteoporosis screening and treatment rates remain low despite guideline recommendations. Crystal Bone is a novel artificial intelligence/machine learning algorithm developed using a large Optum de-identified electronic health record (EHR) dataset to identify patients likely experiencing a fracture within 2 years.

This analysis tested the generalizability of Crystal Bone in three US EHR datasets.

Patients ≥50 years old with ≥2 EHR International Classification of Diseases (ICD) codes, ≥2 years of consecutive EHR history, and ≥4 years of database time since their first EHR ICD code in Optum Care Reliant Medical Group dataset (December 2014 – November 2020), Stanford Health Care dataset (January 2010 – August 2021), and Intermountain Health dataset (January 2012 – May 2022) were included. The primary outcome was area under the receiver operating characteristic (AUROC) for fracture prediction.

Eligible patients (n=106,328) in Reliant and a random test subset in Stanford (n=25,668) and Intermountain Health (n=43,000) were scored by Crystal Bone. AUROC ranged from 0.74 to 0.77 across datasets. As a pre-screening tool providing adjunctive information if further patient review/follow-up is needed, Crystal Bone, at fracture risk threshold of ~0.15 was associated with positive predictive value across datasets of 18%–26%, at which the burden of screening/follow-up versus fracture offset seemed reasonable; negative predictive value (96%–98%) and specificity (97%–99%) were high. Sensitivity (16%–21%) was similar to other fracture prediction models. Among 3,715 patients with Crystal Bone scores above 0.15 in all datasets, 38%–62% had no prior osteoporosis intervention and 8%–23% had no fracture history.

Applying Crystal Bone to EHR data successfully identified patients at risk of fracture within 2 years, including those not detected previously. Automated fracture risk prediction by Crystal Bone demonstrated consistent accuracy and precision across three different US healthcare systems, supporting the generalizability of the algorithm.