A Case Study for the Interrater Reliability of Journal Referees

Abstract

Purpose: Current research indicates problems are emerging in our current system of anonymous journal reviews. We find a proliferation of predatory journals and difficulty in identifying them and explaining the dangers of publishing them to scholars. Predatory journals claim to offer anonymous reviews, but often they do not. Methods: We employ data from The International Journal of Social Work Values and Ethics as a model for transparency ( n = 246). Interrater reliability correlations and chi square was used to contrast data from our literature review. Results: Although the interrater correlations were deemed moderate, our results were better than those in the baseline found in the review of literature. This type of data addressing journals is not available to the scientific community but should be. Conclusion: Based on the synthesis of the review of literature and our case study, eight recommendations are offered to improve the anonymous review process for academic journals.