Module robotparser::model
source · [−]Expand description
Contains models of robots.txt file.
Structs
A model of the robots.txt file that was downloaded over the network.
This model takes into account HTTP response codes when loading the robots.txt file.
To work with this model you should use the trait robotparser::service::RobotsTxtService.
To create this structure you should use the robotparser::parser::parse_fetched_robots_txt.
The model of limiting the frequency of requests to the server.
It’s set by the Request-Rate directive.
The robots.txt model that was obtained after parsing the text of the robots.txt file.
To work with this model you should use the trait robotparser::service::RobotsTxtService.
To create this structure you should use the robotparser::parser::parse_robots_txt.