Struct GenerateOrchestrator 

pub struct GenerateOrchestrator<S: Scraper> { /* private fields */ }

Orchestrates parallel scraping with adaptive concurrency.

Uses a semaphore to limit concurrent scrape operations and supports progress reporting through callbacks. Can adaptively reduce concurrency on rate limit errors (429 responses).

Example

use blz_cli::generate::{GenerateOrchestrator, UrlWithLastmod};
use std::sync::Arc;

// With mock scraper for testing
let scraper = MockScraper::new();
let orchestrator = GenerateOrchestrator::new(scraper, 5)
    .with_progress(|completed, total| {
        println!("Progress: {}/{}", completed, total);
    });

let urls = vec![
    UrlWithLastmod::new("https://example.com/a".to_string()),
    UrlWithLastmod::new("https://example.com/b".to_string()),
];

let results = orchestrator.scrape_all(&urls).await;

Implementations

impl<S: Scraper> GenerateOrchestrator<S>

pub fn new(scraper: S, concurrency: usize) -> Self

Create with specified concurrency.

Arguments

• scraper - The scraping backend to use
• concurrency - Maximum concurrent scrape operations (clamped to 1-50)

pub fn with_default_concurrency(scraper: S) -> Self

Create with default concurrency (5).

pub fn with_progress<F>(self, callback: F) -> Self

where F: Fn(usize, usize) + Send + Sync + 'static,

Set progress callback.

The callback receives (completed, total) after each URL is processed.

pub async fn scrape_all(&self, urls: &[UrlWithLastmod]) -> ScrapeResults

Scrape all URLs in parallel.

Uses a semaphore to limit concurrent operations to the configured concurrency level. Reports progress through the callback if set.

Returns

Aggregated results with successful pages and failed pages.

pub const fn concurrency(&self) -> usize

Get current concurrency level.

pub const fn min_concurrency(&self) -> usize

Get minimum concurrency level.