Struct Web2llmConfig 

pub struct Web2llmConfig {
    pub user_agent: String,
    pub timeout: Duration,
    pub block_private_hosts: bool,
    pub sensitivity: f32,
    pub robots_check: bool,
    pub rate_limit: u32,
    pub max_concurrency: usize,
}

User-facing configuration for the web2llm pipeline. Controls fetch behavior and request identity. Use Web2llmConfig::default() for sensible defaults.

Fields

user_agent: String

The user-agent string sent with every HTTP request. Also used for robots.txt compliance checks.

timeout: Duration

Maximum time to wait for a response before giving up.

block_private_hosts: bool

If true, requests to private, loopback, and link-local addresses are rejected during pre-flight validation. This prevents SSRF attacks when web2llm is used in a service that accepts user-supplied URLs.

Set to false if you need to fetch from localhost or internal hosts in a trusted environment, such as local development or testing.

Defaults to true.

sensitivity: f32

Controls how aggressively secondary content is filtered. A value of 0.1 keeps everything within 10x of the best scoring branch. A value of 0.5 keeps only branches close to the best. Defaults to 0.1.

robots_check: bool

If true, the pipeline will fetch and respect robots.txt before downloading the target page. Defaults to true.

rate_limit: u32

The maximum number of requests allowed per second. Defaults to 5.

max_concurrency: usize

The maximum number of concurrent requests allowed across the whole pipeline. Defaults to 10.

Implementations

impl Web2llmConfig

pub fn new( user_agent: String, timeout: Duration, block_private_hosts: bool, sensitivity: f32, rate_limit: u32, max_concurrency: usize, ) -> Self

Creates a new Web2llmConfig with the specified values.

pub fn with_robots_check(self, check: bool) -> Self

Builder-style method to set whether to check robots.txt.

Trait Implementations

impl Default for Web2llmConfig

fn default() -> Self

Returns the “default value” for a type.