Comparative Performance Evaluation of Keyword and Semantic Search Engines using Different Query Set Categories

Abstract

Background: Keyword search engines are unable to understand the intention of user as a result they produce enormous results for user to distinguish between relevant and non relevant answers of user queries. This has led to rise in requirement to study search capabilities of different search engines. In this research work, experimental evaluation is done based on different metrics to distinguish different search engines on the basis of type of query that can be handled by them. Methods: To check the semantics handling performance, four types of query sets consisting of 20 queries of agriculture domain are chosen. Different query set are single term queries, two term queries, three term queries and NLP queries. Queries from different query set were submitted to Google, DuckDuckGo and Bing search engines. Effectiveness of different search engines for different nature of queries is experimented and evaluated in this research using Grade relevance measures like Cumulative Gain, Discounted Cumulative Gain, Ideal Discounted Cumulative Gain, and Normalized Discounted Cumulative Gain in addition to the precision metric. Results: Our experimental results demonstrate that for single term query, Google retrieves more relevant documents and performs better and DuckDuckGo retrieves more relevant documents for NLP queries. Conclusion: Analysis done in this research shows that DuckDuckGo understand human intention and retrieve more relevant result, through NLP queries as compared to other search engines.