Evaluation of JATSdecoder as an automated text extraction tool for statistical results in scientific reports

Abstract

AbstractThe extraction of statistical results in scientific reports is beneficial for checking studies on plausibility and reliability. The R package JATSdecoder supports the application of text mining approaches to scientific reports. Its function get.stats() extracts all reported statistical results from text and recomputes p values for most standard test results. The output can be reduced to results with checkable or computable p values only. In this article, get.stats()’s ability to extract, recompute and check statistical results is compared to that of statcheck, which is an already established tool. A manually coded data set, containing the number of statistically significant results in 49 articles, serves as an initial indicator for get.stats()’s and statcheck’s differing detection rates for statistical results. Further 13,531 PDF files by 10 mayor psychological journals, 18,744 XML documents by Frontiers of Psychology and 23,730 articles related to psychological research and published by PLoS One are scanned for statistical results with both algorithms. get.stats() almost replicates the manually extracted number of significant results in 49 PDF articles. get.stats() outperforms the statcheck functions in identifying statistical results in every included journal and input format. Furthermore, the raw results extracted by get.stats() increase statcheck’s detection rate. JATSdecoder’s function get.stats() is a highly general and reliable tool to extract statistical results from text. It copes with a wide range of textual representations of statistical standard results and recomputes p values for two- and one-sided tests. It facilitates manual and automated checks on consistency and completeness of the reported results within a manuscript.