Design of topic Web crawler based on improved PageRank algorithm

Abstract

Abstract With the continuous development of network information technology, the network is filled with a large number of all kinds of unstructured data called big data. However, this data is not easily stored in a local database. People realize that it is essential to get useful information from the Internet efficiently. The effort to gather information by human hands has led to the emergence of web crawler technology. However, the existing search engines still have shortcomings in topic similarity judgment and web page sorting algorithm. Therefore, this paper applies PageRank algorithm to topic crawler, constructs a vertical search engine, and introduces topic relevance factor to suppress "topic drift" according to the shortcomings of PageRank algorithm.