Evaluation of diagnostic and triage accuracy and usability of a symptom checker in an emergency department: observational study (Preprint)

Abstract

Symptom checkers (SCs) are diagnostic decision support apps for patients, used by tens of millions of people annually. They are designed to provide diagnosis and triage advice and assist users in seeking the appropriate level of care. Little evidence is available on their diagnostic and triage accuracy with direct use by patients for urgent conditions.

To determine the diagnostic and triage accuracy and usability of a symptom checker in use by for patients presenting to an emergency department

We recruited a convenience sample of English-speaking patients presenting for care in an urban emergency department. Each consenting patient used a leading SC from Ada Health prior to ED evaluation. Diagnostic accuracy was evaluated, comparing (1) the SC’s diagnoses and (2) those of 3 independent emergency physicians viewing the patient-entered symptom data, to (3) the final diagnoses from the ED evaluation. The Ada diagnoses and triage were also critiqued by the independent physicians. Patients completed a usability survey based on the Technology Acceptance Model.

Forty participants (80% of those approached, mean age 40.1 years (range 18 – 76); 65% female, 68% White, 48% Hispanic or Latino, completed the SC assessment and usability survey. Sensitivity for at least one of the ED final diagnoses by Ada (based on it’s top 5 diagnoses) was 70.0% (95% CI 54% – 86%) close to the mean sensitivity for the 3 physicians (on their top 3 diagnoses) of 68.9%. The physicians rated Ada triage decisions as 62% fully agree, and 24% safe but too cautious. It was rated unsafe and too risky in 22% by at least one physician, 14% by two and 5% by all 3, although none of the under-triaged patients suffered an adverse event in the ED. Some cases only had symptoms recorded for their ED diagnosis, limiting analysis. Usability was rated highly, participants agreed or strongly agreed a mean of 84.6% with the 7 TAM usability questions, although “satisfaction” and “enjoyment” were rated low.

This study provides preliminary evidence that a SC can provide acceptable usability and diagnostic accuracy for patients with a variety of urgent conditions. 14% of SC triage recommendations were deemed unsafe and too risky by at least 2 physicians based on the symptoms recorded. Larger studies are needed of diagnosis and triage performance with direct patient use in different clinical environments.