Expand description
The Crawly web crawler efficiently fetches and stores content from web pages.
It respects robots.txt guidelines and handles rate limits.
Structsยง
- Crawler
- Main structure for the
Crawlercontaining necessary utilities and caches. - Crawler
Builder - Builder pattern for
Crawler. Allows for customizable configurations. - Mime
- A parsed mime or media type.