Struct SitemapRequestListBuilder 

pub struct SitemapRequestListBuilder { /* private fields */ }

Configures a streaming SitemapRequestList.

Implementations

impl SitemapRequestListBuilder

pub fn sitemap_url(self, url: Url) -> Self

Adds one root sitemap URL.

pub fn sitemap_urls(self, urls: impl IntoIterator<Item = Url>) -> Self

Adds root sitemap URLs.

pub fn http_client(self, client: Arc<dyn HttpClient>) -> Self

Sets the HTTP backend used to stream sitemap documents.

pub fn persist( self, kvs: Arc<dyn KeyValueStore>, key: impl Into<String>, ) -> Self

Enables persisted progress under key.

An in-progress sitemap is fetched again from byte zero after restart, and its already-emitted entries are skipped. This requires stable sitemap ordering across fetches, which is the practical norm for sitemap files.

pub fn label(self, label: impl Into<String>) -> Self

Applies a routing label to every emitted request.

pub fn user_data(self, user_data: UserData) -> Self

Applies user data to every emitted request.

pub fn limit(self, limit: u64) -> Self

Limits the total number of emitted requests.

pub fn build(self) -> Result<SitemapRequestList, CrawlError>

Builds a request list without performing network or storage I/O.

Trait Implementations

impl Default for SitemapRequestListBuilder

fn default() -> SitemapRequestListBuilder

Returns the “default value” for a type.