Crate crawly

source ·
Expand description

The Crawly web crawler efficiently fetches and stores content from web pages. It respects robots.txt guidelines and handles rate limits.

Structs

  • Crawler
    Main structure for the Crawler containing necessary utilities and caches.
  • CrawlerBuilder
    Builder pattern for Crawler. Allows for customizable configurations.
  • Mime
    A parsed mime or media type.