Fifty Ways to Tag your Pubtypes: Multi-Tagger, a Set of Probabilistic Publication Type and Study Design Taggers to Support Biomedical Indexing and Evidence-Based Medicine

Abstract

AbstractObjectiveIndexing articles according to publication types (PTs) and study designs can be a great aid to filtering literature for information retrieval, especially for evidence syntheses. In this study, 50 automated machine learning based probabilistic PT and study design taggers were built and applied to all articles in PubMed.Materials and MethodsPubMed article metadata from 1987-2014 were used as training data, with 2015 used for recalibration. The set of articles indexed with a particular study design MeSH term or PT tag was used as positive training sets. For each PT, the rest of the literature from the same time period was used as its negative training set. Multiple features based on each article title, abstract and metadata were used in training the models. Taggers were evaluated on PubMed articles from 2016 and 2019. A manual analysis was also performed.ResultsOf the 50 predictive models that we created, 44 of these achieved an AUC of ∼0.90 or greater, with many having performance above 0.95. Of the clinically related study designs, the best performing was SYSTEMATIC_REVIEW with an AUC of 0.998; the lowest performing was RANDOM_ALLOCATION, with an AUC of 0.823.DiscussionThis work demonstrates that is feasible to build a large set of probabilistic publication type and study design taggers with high accuracy and ranking performance. Automated tagging permits users to identify qualifying articles as soon as they are published, and allows consistent criteria to be applied across different bibliographic databases. Probabilistic predictive scores are more flexible than binary yes/no predictions, since thresholds can be tailored for specific uses such as high recall literature search, user-adjustable retrieval size, and quality improvement of manually annotated databases.ConclusionThe PT predictive probability scores for all PubMed articles are freely downloadable at http://arrowsmith.psych.uic.edu/evidence_based_medicine/mt_download.html for incorporation into user tools and workflows. Users can also perform PubMed queries at our Anne O’Tate value-added PubMed search engine http://arrowsmith.psych.uic.edu/cgi-bin/arrowsmith_uic/AnneOTate.cgi and filter retrieved articles according to both NLM-annotated and model-predicted publication types and study designs.