List of all items

Structs

• cache::CacheStats
• cache::CacheStatsSnapshot
• cache::RobotsCache
• checker::RobotsChecker
• checker::RobotsDiagnostics
• fetcher::FetchStats
• fetcher::FetchStatsSnapshot
• fetcher::RobotsFetcher
• matcher::RobotsMatcher
• parser::RobotsParser
• types::CompiledPattern
• types::Decision
• types::EffectiveRules
• types::Group
• types::RequestRate
• types::RobotsCacheKey
• types::RobotsConfig
• types::RobotsPolicy
• types::Rule

Enums

• types::DecisionReason
• types::FetchStatus
• types::RuleKind

Functions

• cache::parse_cache_control
• matcher::get_crawl_delay
• matcher::is_allowed
• parser::encoding::normalize_for_comparison
• parser::encoding::normalize_path_for_matching

Constants

• cache::DEFAULT_CACHE_TTL
• cache::MAX_CACHE_TTL
• parser::MAX_ROBOTS_SIZE