Struct CrawlerDetector 

pub struct CrawlerDetector { /* private fields */ }

Bot and crawler detector.

Implementations

impl CrawlerDetector

pub async fn new(config: CrawlerConfig) -> Result<Self, String>

Create a new crawler detector.

pub fn disabled() -> Self

Create a disabled crawler detector when initialization fails.

pub async fn verify( &self, user_agent: &str, client_ip: IpAddr, ) -> CrawlerVerificationResult

Verify a request’s crawler status.

Security

• Validates user_agent length to prevent ReDoS
• Applies DNS failure policy for fail-secure behavior
• Bounds stats map sizes to prevent memory exhaustion

pub fn check_bad_bot( &self, user_agent: &str, ) -> Option<&'static BadBotSignature>

Check if user agent matches a bad bot signature.

Security

Exclusion logic is handled in code (not regex) to prevent ReDoS attacks from complex negative lookaheads.

pub fn stats(&self) -> CrawlerStatsSnapshot

Get statistics snapshot.

pub fn is_enabled(&self) -> bool

Check if the detector is enabled.

pub fn config(&self) -> &CrawlerConfig

Get the configuration.

pub fn should_block_bad_bots(&self) -> bool

Check if bad bots should be blocked.

pub fn get_crawler_distribution(&self, limit: usize) -> Vec<(String, u64)>

Returns the distribution of crawler hits by name.

pub fn get_bad_bot_distribution(&self, limit: usize) -> Vec<(String, u64)>

Returns the distribution of bad bot hits by signature.

Trait Implementations

impl CrawlerDetection for CrawlerDetector

Implement the CrawlerDetection trait for CrawlerDetector

fn verify<'life0, 'life1, 'async_trait>( &'life0 self, user_agent: &'life1 str, client_ip: IpAddr, ) -> Pin<Box<dyn Future<Output = CrawlerVerificationResult> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Verify a request’s crawler status.

fn is_enabled(&self) -> bool

Check if the detector is enabled.

fn should_block_bad_bots(&self) -> bool

Check if bad bots should be blocked.

fn stats(&self) -> CrawlerStatsSnapshot

Get statistics snapshot.