Incremental Refinement of Relevance Rankings: Introducing a New Method Supported with Pennant Retrieval

Abstract

Purpose: Relevance ranking algorithms rank retrieved documents based on the degrees of topical similarity (relevance) between search queries and documents. This paper aims to introduce a new relevance ranking method combining a probabilistic topic modeling algorithm with the “pennant retrieval” method using citation data. Data and Method: We applied this method to the iSearch corpus consisting of c. 435,000 physics papers. We first ran the topic modeling algorithm on titles and summaries of all papers for 65 search queries and obtained the relevance ranking lists. We then used the pennant retrieval to fuse the citation data with the existing relevance rankings, thereby incrementally refining the results. The outcome produced better relevance rankings with papers covering various aspects of the topic searched as well as the more marginal ones. The Maximal Marginal Relevance (MMR) algorithm was used to evaluate the retrieval performance of the proposed method by finding out its effect on relevance ranking algorithms that we used. Findings: Findings suggest that the terms used in different contexts in the papers might sometimes be overlooked by the topic modeling algorithm. Yet, the fusion of citation data to relevance ranking lists provides additional contextual information, thereby further enriching the results with diverse (interdisciplinary) papers of higher relevance. Moreover, results can easily be re-ranked and personalized. Implications: We argue that once it is tested on dynamic corpora for computational load, robustness, replicability, and scalability, the proposed method can in time be used in both local and international information systems such as TR-Dizin, Web of Science, and Scopus. Originality: The proposed method is, as far as we know, the first one that shows that relevance rankings produced with a topic modeling algorithm can be incrementally refined using pennant retrieval techniques based on citation data.