Affiliation:
1. Wuhan Polytechnic University
Abstract
The web data collection is the process of collecting the semi-structured, large-scale and redundant data which include web content, web structure and web usage in the web by the crawler and it is often used for the information extraction, information retrieval, search engine and web data mining. In this paper, the web data collection principle is introduced and some related topics are discussed such as page download, coding problem, updated strategy, static and dynamic page. The multithread technology is described and multithread mode for the web data collection is proposed. The web data collection with multithread can get better resource utilization, better average response time and better performance.
Publisher
Trans Tech Publications, Ltd.
Reference10 articles.
1. Christopher Olston and Marc Najork, Web Crawling, Foundations and Trends in Information Retrieval, (2010).
2. Carlos Castillo and Ricardo Baeza-Yates. A new crawling model. In Poster proceedings ofthe eleventh conference on World Wide Web, Honolulu, Hawaii, USA, May (2002).
3. Soderland, S., Learning information extraction rules for semi-structured and free text. Journal of Machine Learning, 34(1-3): 233-272, (1999).
4. Freitag, D., Information extraction from HTML: Application of a general learning approach. Proceedings of the Fifteenth Conference on Artificial Intelligence, (1998).
5. Bing Liu., Web Data Mining: Exploring Hyperlinks, Contents and Usage Data, Springer, (2009).