Struct robotparser::model::FetchedRobotsTxt
source · [−]pub struct FetchedRobotsTxt { /* private fields */ }
Expand description
A model of the robots.txt file that was downloaded over the network.
This model takes into account HTTP response codes when loading the robots.txt file.
To work with this model you should use the trait robotparser::service::RobotsTxtService
.
To create this structure you should use the robotparser::parser::parse_fetched_robots_txt
.
Implementations
sourceimpl FetchedRobotsTxt
impl FetchedRobotsTxt
sourcepub fn get_fetched_at(&self) -> &SystemTime
pub fn get_fetched_at(&self) -> &SystemTime
Returns the system time when the robots.txt file was downloaded over the network.
Trait Implementations
sourceimpl Clone for FetchedRobotsTxt
impl Clone for FetchedRobotsTxt
sourcefn clone(&self) -> FetchedRobotsTxt
fn clone(&self) -> FetchedRobotsTxt
Returns a copy of the value. Read more
1.0.0 · sourcefn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
Performs copy-assignment from source
. Read more
sourceimpl Debug for FetchedRobotsTxt
impl Debug for FetchedRobotsTxt
sourceimpl RobotsTxtService for FetchedRobotsTxt
impl RobotsTxtService for FetchedRobotsTxt
sourcefn can_fetch(&self, user_agent: &str, url: &Url) -> bool
fn can_fetch(&self, user_agent: &str, url: &Url) -> bool
Using the parsed robots.txt decide if useragent can fetch url.
sourcefn get_crawl_delay(&self, user_agent: &str) -> Option<Duration>
fn get_crawl_delay(&self, user_agent: &str) -> Option<Duration>
Returns the crawl delay for this user agent as a Duration, or None if no crawl delay is defined.
sourcefn normalize_url(&self, url: &mut Url) -> bool
fn normalize_url(&self, url: &mut Url) -> bool
Removes the request parameters from the url that were listed in the Clean-param
directive.
This method CHECKS that the origin of the transmitted url matches the origin of robots.txt.
Returns true if the operation was applied to the passed url.
In other cases it returns false. Read more
sourcefn normalize_url_ignore_origin(&self, url: &mut Url)
fn normalize_url_ignore_origin(&self, url: &mut Url)
Removes the request parameters from the url that were listed in the Clean-param
directive.
This method DOES NOT CHECK that the origin of the transmitted url coincides with the origin of robots.txt. Read more
sourcefn get_sitemaps(&self) -> &[Url]
fn get_sitemaps(&self) -> &[Url]
Returns the list of URL sitemaps that have been listed in the robots.txt file.
sourcefn get_req_rate(&self, user_agent: &str) -> Option<RequestRate>
fn get_req_rate(&self, user_agent: &str) -> Option<RequestRate>
Returns information about the restrictions set for sending HTTP requests to the server.
Auto Trait Implementations
impl RefUnwindSafe for FetchedRobotsTxt
impl Send for FetchedRobotsTxt
impl Sync for FetchedRobotsTxt
impl Unpin for FetchedRobotsTxt
impl UnwindSafe for FetchedRobotsTxt
Blanket Implementations
sourceimpl<T> BorrowMut<T> for T where
T: ?Sized,
impl<T> BorrowMut<T> for T where
T: ?Sized,
const: unstable · sourcefn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
sourceimpl<T> Instrument for T
impl<T> Instrument for T
sourcefn instrument(self, span: Span) -> Instrumented<Self>
fn instrument(self, span: Span) -> Instrumented<Self>
sourcefn in_current_span(self) -> Instrumented<Self>
fn in_current_span(self) -> Instrumented<Self>
sourceimpl<T> ToOwned for T where
T: Clone,
impl<T> ToOwned for T where
T: Clone,
type Owned = T
type Owned = T
The resulting type after obtaining ownership.
sourcefn clone_into(&self, target: &mut T)
fn clone_into(&self, target: &mut T)
toowned_clone_into
)Uses borrowed data to replace owned data, usually by cloning. Read more
sourceimpl<T> WithSubscriber for T
impl<T> WithSubscriber for T
sourcefn with_subscriber<S>(self, subscriber: S) -> WithDispatch<Self> where
S: Into<Dispatch>,
fn with_subscriber<S>(self, subscriber: S) -> WithDispatch<Self> where
S: Into<Dispatch>,
Attaches the provided Subscriber
to this type, returning a
WithDispatch
wrapper. Read more
sourcefn with_current_subscriber(self) -> WithDispatch<Self>
fn with_current_subscriber(self) -> WithDispatch<Self>
Attaches the current default Subscriber
to this type, returning a
WithDispatch
wrapper. Read more