halldyll-core 0.1.0

Core scraping engine for Halldyll - high-performance async web scraper for AI agents
Documentation
1
2
3
4
5
6
7
//! Politeness - Robots.txt and throttling

pub mod robots;
pub mod throttle;

pub use robots::{RobotsChecker, RobotsCache};
pub use throttle::{DomainThrottler, ThrottleState};