Struct Fetcher 

pub struct Fetcher { /* private fields */ }

HTTP client for fetching llms.txt documentation with conditional request support

Implementations

impl Fetcher

pub fn new() -> Result<Self>

Creates a new fetcher with configured HTTP client

pub fn with_timeout(timeout: Duration) -> Result<Self>

Creates a new fetcher with a custom request timeout (primarily for tests)

pub async fn fetch_with_cache( &self, url: &str, etag: Option<&str>, last_modified: Option<&str>, ) -> Result<FetchResult>

Fetches a URL with conditional request support using ETag and Last-Modified headers

pub async fn fetch(&self, url: &str) -> Result<(String, String)>

Fetches a URL without conditional request support, returning content and SHA256 hash

pub async fn head_metadata(&self, url: &str) -> Result<HeadInfo>

Perform a HEAD request to retrieve basic metadata for a URL without downloading content