The limits of inductive reasoning for clinical evidence appraisal – a simulation study (Preprint)

Abstract

Abstract Objectives To establish the possible likelihood of a body of evidence, inductively judged to be of ‘low bias risk’/‘high-quality’ according to a limited set of appraisal criteria, of actually being error-free. Methods A total of 45 simulation trials were generated and randomly assigned to 0–5 errors out of a total of 65 error domains. The trials were then appraised for errors with a simulated appraisal tool consisting of five pre-specified error domains. Trial appraisal yielded either true positive, true negative, false negative or false positive results. From these values, the negative likelihood ratio (–LR) with 95% confidence interval (CI) was computed. –LR computation was repeated 25 times, each with newly generated random values for all 45 trials. The individual results of all 25 runs were statistically pooled. The pooled –LR result with 95% CI was interpreted as how likely a ‘low bias risk’/‘high-quality’ rated body of evidence is actually error-free. Results The pooled –LR was 0.84 (95% CI: 0.80–0.88, I2 = 0.0%). The result suggests that error-free evidence is only 1.2 times more likely to be rated as ‘low bias risk’/‘high-quality’ than evidence containing some form of error. Conclusion The likelihood of a ‘low bias risk’/‘high-quality’ rated body of evidence being actually error-free is small and the inductive generalisation from any limited, pre-specified set of appraisal criteria rarely justifies a high level of confidence that a ‘low bias risk’/‘high-quality’ rating of clinical evidence reflects the true effect of a certain treatment without being affected by error.