pagesift-frontier 1.1.2

pagesift crawl frontier and focused-crawl scoring: deduplication, URL traps, link scoring, stop rules.
Documentation