Crate siteforge 

Website archiving primitives and the high-level Siteforge facade.

siteforge archives websites into file-first, AI-readable local knowledge archives. The CLI and TUI are built on the same public modules exposed here; programmatic callers can use Siteforge for the common archive, resume, search, export, pack, and verify flows without manually stitching modules together.

Completed archives include readable Markdown, structured JSON, JSONL chunks, agent entrypoints, compact crawl metadata, BLAKE3 checksums, and a thesa-compatible .thesa/ sidecar for external verification.

Example

use siteforge::{Config, Siteforge};

#[tokio::main]
async fn main() -> siteforge::Result<()> {
    let mut config = Config::default();
    config.archive_root = "archives".into();

    let siteforge = Siteforge::new(config);
    let summary = siteforge.archive_page("https://example.com/").await?;

    println!("{}", summary.archive_path.display());
    Ok(())
}

Re-exports

pub use archive::ArchiveLayout;

pub use archive::ArchiveManifest;

pub use archive::CrawlStats;

pub use archive::ThesaSidecarSummary;

pub use config::ArchiveOutputProfile;

pub use config::Config;

pub use crawler::ArchiveProgress;

pub use crawler::ArchiveProgressPhase;

pub use crawler::ArchiveProgressSender;

pub use crawler::ArchiveRunOptions;

pub use crawler::ArchiveSummary;

pub use errors::Result;

pub use errors::SiteforgeError;

pub use export::ExportFormat;

pub use storage::SearchResult;

pub use storage::StoredAsset;

pub use storage::StoredPage;

Modules

archive

cli

config

crawler

errors

export

fetch

ocr

parse

render

search

storage

tui

Structs

Siteforge

High-level programmatic interface over siteforge configuration and archive operations.