Skip to main content

Module generate

blz_cli

Module generate 

Source
Expand description

Generate command orchestration for creating llms-full.txt files.

This module provides the orchestration layer for the blz generate command, which scrapes discovered URLs via Firecrawl and assembles them into a complete llms-full.txt file.

§Key Components

§Example

use blz_cli::generate::{GenerateOrchestrator, UrlWithLastmod, ScrapeResults};

// URLs would come from sitemap discovery
let urls = vec![
    UrlWithLastmod::new("https://example.com/docs/intro".to_string()),
    UrlWithLastmod::new("https://example.com/docs/api".to_string()),
];

// Create orchestrator (would use real FirecrawlCli)
// let cli = FirecrawlCli::detect().await?;
// let orchestrator = GenerateOrchestrator::new(cli, 5)
//     .with_progress(|completed, total| {
//         println!("Progress: {}/{}", completed, total);
//     });
//
// let results = orchestrator.scrape_all(&urls).await;
// println!("Successful: {}, Failed: {}", results.successful.len(), results.failed.len());

Structs§

GenerateOrchestrator
Orchestrates parallel scraping with adaptive concurrency.
ScrapeError
Error from a scrape operation.
ScrapeResult
Result of a scrape operation.
ScrapeResults
Results of a scraping operation.
UrlWithLastmod
URL with optional lastmod for change detection.

Traits§

Scraper
Trait for scraping URLs (allows mocking in tests).

Type Aliases§

ProgressCallback
Progress callback type for reporting scrape progress.