Randomized Clinical Trials can be Accurately Classified in terms of their Methodological Validity by their Representations and Cochrane Reviewers’ Comments (Preprint)

Abstract

Evidence-based medicine requires evaluating and critically appraising documents. The process needs (semi)automated methods to distinguish valid from invalid randomized clinical trials (RCTs).

The present study aimed to test the potential of RCT representations, Cochrane reviewers’ comments, and comment-expanded representations to learn a machine to classify RCTs into different validity categories.

To build a test collection, 9,063 RCTs referred to by 132 open-access systematic reviews in the Cochrane database were identified, being tagged as “included” and having PMID. The RCTs’ methodological validity scores were extracted from the Cochrane database and their representative features (i.e., titles, abstracts, and author keywords) from PubMed. To classify the RCTs based on their (in)validity, machine learning was conducted using the kNN algorithm, with 10-fold cross-validation, in the KNIME Data Mining Platform.

The results showed that the comments and representations can accurately classify RCTs. The classification accuracy values for the comments ranged from 0.758 (for incomplete outcome data) to 0.896 (for potential threats to validity). The accuracy values for the RCT representations ranged from 0.691 (for random sequence generation) to 0.757 (for selective outcome reporting). After expanding the representations by the comments, improvements were observed in the accuracy values, ranging from 0.105 (for blinding of participants, personnel, and outcome assessors) to 0.196 (for random sequence generation).

The RCTs’ representations and Cochrane reviewers’ comments showed effectiveness in classifying them in their validity categories, with the latter being even more powerful. The comments also enriched the representations and improved their classification accuracy.