Struct RobotsParser 

pub struct RobotsParser { /* private fields */ }

Parser for robots.txt files

Implementations

impl RobotsParser

pub fn new() -> Self

Create a new parser with default settings

pub fn with_max_size(max_size: usize) -> Self

Create a parser with custom max size

pub fn parse(&self, content: &str, ttl: Duration) -> RobotsPolicy

Parse robots.txt content into a policy

Trait Implementations

impl Default for RobotsParser

fn default() -> Self

Returns the “default value” for a type.