Expand description
Contains models of robots.txt file.
Structs§
- Error
- Fetched
Robots Txt - A model of the robots.txt file that was downloaded over the network.
This model takes into account HTTP response codes when loading the robots.txt file.
To work with this model you should use the trait
robotparser::service::RobotsTxtService. To create this structure you should use therobotparser::parser::parse_fetched_robots_txt. - Request
Rate - The model of limiting the frequency of requests to the server.
It’s set by the
Request-Ratedirective. - Robots
Txt - The robots.txt model that was obtained after parsing the text of the robots.txt file.
To work with this model you should use the trait
robotparser::service::RobotsTxtService. To create this structure you should use therobotparser::parser::parse_robots_txt.