Research on focused crawler based upon network topology

Abstract

Subject-oriented focued crawler, skips irrelevant links, and receives only relevant information. However, general fouced crawler couldn,t deal with redundant links. This paper presents a kind of focused crawler based upon network topology. The crawler gets original URL sets from search engine, then calculates content similarity by the model of vector space. It analyzes link structure of websites, moreover modifies weight of URL according to the characteristic of scale-free network. Relevance feedback is used to disengage irrelevant regions, and the length of buffer is set for irrelevant URL by the distance between URL and seed pages. Experiments results prove that the precision of this focused crawler is higher than general crawler. Show more