Module robotparser::http
source · [−]Expand description
Request builder & response parsers for other http libraries.
Supported libraries
To enable support for the required library, you need to add this feature to your Cargo.toml.
Now only one library is supported - reqwest.
But you can also add support for other libraries.
Modules
Support for reqwest library.
Constants
User agent of this crate.
Traits
Trait to fetch and parse the robots.txt file. Must be implemented on http-client.