Research and Design of Web Crawler for Music Resources Finding

Abstract

This paper designs an automatic web crawler system which crawls music resources on the Internet. Firstly, this paper gives the architecture of the system and the function of each module; then describes the detailed design of each module; Finally, the key technologies and algorithms used in the system are given in a detailed description, including the use of χ2 statistics to select feature words, TF-IDF algorithm to calculate the weights of feature words, the correlation of web page and music theme using vector space model.