Using Transformer-based Language Models to Identify Publications from Clinical Trials that Use Nested Designs (Preprint)

Abstract

For the public health community, monitoring recently published articles is crucial for staying informed about the latest research developments and advancements. However, identifying publications with specific research designs from the extensive body of public health publications is a challenge with currently available methods. Using search queries to retrieve publications of this type often yields low sensitivity, failing to identify many qualifying publications. While document classification techniques like classical and deep learning machine learning models have been widely used for categorizing biomedical documents, the use of state-of-the-art language models for this purpose has yet to be explored. The language models pre-trained on biomedical data that do exist have suboptimal performance in predicting categories in specialized tasks like identifying publications from clinical trials that use nested designs.

Our objective is to develop a fine-tuned pre-trained language model that can accurately identify publications from clinical trials that use a Group- or Cluster-Randomized Trial (GRT), Individually Randomized Group-Treatment Trial (IRGT), or Stepped Wedge Group- or Cluster-Randomized Trial (SWGRT) design within the biomedical literature.

We fine-tuned the BiomedBERT language model using a dataset of biomedical literature from the Office of Disease Prevention at the National Institutes of Health. The model was trained to classify publications into three categories of clinical trials that use nested designs. The model performance was evaluated on unseen data and demonstrated high sensitivity and specificity for each class.

When our proposed model was tested for generalizability with unseen data, it delivered high sensitivity and specificity for each class as follows: Non-randomized trials (0.95 and 0.93), GRTs (0.94 and 0.90), IRGTs (0.81 and 0.97), and SWGRTs (0.96 and 0.99), respectively.

This model offers a valuable tool for the public health community to directly identify publications from clinical trials that use one of three classes of nested designs.