Applying multi-objective particle swarm optimization-based dynamic adaptive hyperlink evaluation to focused crawler for meteorological disasters

Abstract

Abstract In traditional semantic-based focused crawlers, the topical priority of unvisited hyperlinks is calculated by linearly integrating pre-defined topical similarity evaluation metrics and their corresponding weighted factors. However, these weighted factors are manually determined by personal experience, which may introduce bias in evaluating unvisited hyperlinks, resulting in topic deviation during crawling. To address this problem, we propose a dynamic adaptive focused crawler, denoted by FCMOPSO, based on multi-objective particle swarm optimization (MOPSO). For topic representation, two domain ontologies of meteorological disasters are constructed. Additionally, we present a comprehensive priority evaluation method (CPEM) of hyperlink concerning both webpage content and hyperlink structure. In MOPSO, the weights of metrics of the CPEM can be updated in every crawling iteration. Furthermore, we utilize the non-dominant sorting with the nearest farthest candidate solution (NS_NFCS) to ensure the diversity of crawling hyperlinks and expand the search range. Compared with focused crawler strategies in the literature, the experimental results on domains of rainstorm disaster and typhoon disaster show that our proposed FCMOPSO achieves satisfactory performance that it can obtain more topic-relevant webpages with ideal time consumption.