Struct RobotsParser

pub struct RobotsParser { /* private fields */ }

Implementations

impl RobotsParser

pub fn new(rules: Vec<Robots>) -> RobotsParser

Creates a new RobotsParser from the given Robots Rules

pub fn parse<'a>(input: &'a str) -> Result<RobotsParser, &'static str>

Parses a robots.txt input string

pub fn parse_path<P: AsRef<Path>>(path: P) -> Result<RobotsParser, &'static str>

Parses a robots.txt file from the given path

pub fn parse_url<U: Into<Url>>(url: U) -> Result<RobotsParser, &'static str>

Parses a robots.txt file from the given url

pub fn can_fetch(&self, agent: &str, path: &str) -> bool

Decides if a path can be fetched by an agent

Trait Implementations

impl Clone for RobotsParser

fn clone(&self) -> RobotsParser

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for RobotsParser

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for RobotsParser

fn eq(&self, other: &RobotsParser) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Eq for RobotsParser

impl StructuralPartialEq for RobotsParser