A Scalable Real-Time Agent-Based Information Retrieval Engine

Abstract

In distributed information retrieval systems, information in web should be ranked based on a combination of multiple features. Linear combination of ranks has been the dominant approach due to its simplicity and efficiency. Such a combination scheme in distributed infrastructure requires that ranks in resources or agents are comparable to each other. The main challenge is how to transform the raw rank values of different criteria appropriately to make them comparable before any combination. In this manuscript, we will demonstrate how to rank Web documents based on its resource-provided information stream and how to combine and incorporate several raking schemas in one time. The system was tested on the queries provided by a Text Retrieval Conference (TREC), and our experimental results showed that it is robust and efficient compared with similar platforms that used offline data resources.