Struct Extensions 

pub struct Extensions<'a> {
    pub sitemaps: Vec<&'a str>,
    pub crawl_delays: Vec<CrawlDelay<'a>>,
    pub hosts: Vec<&'a str>,
    pub clean_params: Vec<CleanParam<'a>>,
    pub other: Vec<Directive<'a>>,
}

Available on crate feature extensions only.

Feature-gated metadata for common non-standard directives.

These values are collected for callers that need them, but they do not affect access decisions returned by RobotsTxt::is_allowed.

Examples

use fast_robots::RobotsTxt;

let robots = RobotsTxt::parse(
    "Sitemap: https://example.com/sitemap.xml\n\
     Host: example.com\n\
     User-agent: *\n\
     Crawl-delay: 10\n",
);

assert_eq!(robots.extensions.sitemaps, ["https://example.com/sitemap.xml"]);
assert_eq!(robots.extensions.hosts, ["example.com"]);
assert_eq!(robots.extensions.crawl_delays[0].value, "10");

Fields

sitemaps: Vec<&'a str>

Sitemap directive values.

crawl_delays: Vec<CrawlDelay<'a>>

Crawl-delay directive values, including the current group agents.

hosts: Vec<&'a str>

Host directive values.

clean_params: Vec<CleanParam<'a>>

Clean-param directive values.

other: Vec<Directive<'a>>

Unknown non-core directives preserved as key/value pairs.

Trait Implementations

impl<'a> Clone for Extensions<'a>

fn clone(&self) -> Extensions<'a>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<'a> Debug for Extensions<'a>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'a> Default for Extensions<'a>

fn default() -> Extensions<'a>

Returns the “default value” for a type.

impl<'a> PartialEq for Extensions<'a>

fn eq(&self, other: &Extensions<'a>) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl<'a> Eq for Extensions<'a>

impl<'a> StructuralPartialEq for Extensions<'a>