Determining the informativeness of comments: a natural language study of F1000Research open peer review reports

Abstract

PurposeSocial comments are rich in information and useful in evaluating, ranking or retrieving different kinds of materials. However, their merits in representing or providing added values to scientific articles have not yet been studied. Therefore, the present study investigates the informativeness of open review reports as a kind of social comments in a scholarly setting.Design/methodology/approachA test collection was built consisting of 100 randomly selected queries, 1,962 reviewed documents and their reviewers' open reports from F1000Research. They were analyzed using natural language techniques. The comments' salient words were compared to the documents' and also to the Medical Subject Headings (MeSH) salient words. The receiver operating characteristic (ROC) curve was used to test the accuracy of the comments in representing their related articles.FindingsThe papers' contents and comments have a considerable number of salient words in common. The comments' salient words are also largely found in the MeSH, signifying their consistency with the knowledge tree and their potential to add some complementary features to their related items. The ROC curves confirm the accuracy of the comments in retrieving their related papers.Originality/valueThis research is the first to reveal the merits of open review reports on scientific papers, in terms of their relatedness to their mother articles, in specific, and to the knowledge tree, in general. They are found informative in not only representing the reviewed papers but also in adding values to the contents of the papers.