Struct RobotsConfig 

pub struct RobotsConfig {
    pub respect: bool,
    pub ignore_domains: Vec<String>,
    pub default_ttl: Duration,
    pub failure_ttl: Duration,
}

Robots.txt fetch + respect knobs.

Fields

respect: bool

ignore_domains: Vec<String>

Hosts for which robots.txt is not fetched and rules are not enforced. Lowercased in-place by validate.

default_ttl: Duration

Used when the robots.txt HTTP response has no Cache-Control: max-age.

failure_ttl: Duration

Used when robots.txt fetch failed with 5xx or transport error (fail-closed). Short by design so a recovered server is picked up quickly.

Trait Implementations

impl Clone for RobotsConfig

fn clone(&self) -> RobotsConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for RobotsConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for RobotsConfig

fn default() -> Self

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for RobotsConfig

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Serialize for RobotsConfig

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.