siteforge

siteforge is a Rust CLI/TUI tool for archiving websites into local, AI-readable knowledge archives. It stores raw snapshots, structured metadata, cleaned Markdown, JSON, JSONL semantic chunks, searchable SQLite metadata, and token-budgeted AI context packs.

The TUI is built with scrin as a hard dependency. Terminal loaders and status effects use aisling through Scrin’s Aisling-backed LoaderPlayer, with direct aisling::LoaderProgress progress data in the dashboard.

Examples

siteforge archive https://catlikecoding.com/unity/tutorials/ --full-site
siteforge archive https://example.com/article
siteforge archive --input urls.txt -j 8
siteforge archive https://catlikecoding.com/unity/tutorials/ --full-site --concurrency 4 --delay-ms 1000
siteforge archive https://docs.example.test --full-site --archive-profile agent
siteforge archive https://docs.example.test --full-site --archive-profile lean
siteforge archive https://docs.example.test/private --header "Authorization: Bearer <token>" --cookie "session=..."
siteforge archive https://app.example.test/docs --render-js --browser-command chromium --browser-profile ~/.config/chromium-siteforge
siteforge archive https://app.example.test/docs --render-js-auto --browser-command chromium
siteforge archive https://example.com/docs --full-site --progress always
siteforge tui
siteforge list
siteforge inspect catlikecoding-20260629120000
siteforge diagnostics --archive-id catlikecoding-20260629120000
siteforge search "mesh deformation code"
siteforge export catlikecoding-20260629120000 --format markdown
siteforge export catlikecoding-20260629120000 --format basic-markdown
siteforge export catlikecoding-20260629120000 --format agent-tar
siteforge export catlikecoding-20260629120000 --format zip
siteforge export catlikecoding-20260629120000 --format tar
siteforge pack catlikecoding-20260629120000 --for-ai --max-tokens 120000
siteforge verify catlikecoding-20260629120000
siteforge resume catlikecoding-20260629120000
siteforge config

Build And Test

cargo install siteforge
cargo fmt
cargo check
cargo clippy
cargo test
cargo run -- archive https://example.com/article
cargo run -- tui

The published Cargo package excludes local archive outputs, session files, logs, .opencode/, and build artifacts.

Basic CLI

Use -j/--concurrency/--jobs to control how many page fetches can run concurrently. Defaults are tuned for a modest crawl pace, and per-origin politeness still applies through configured delay, robots crawl-delay, and robots request-rate.

siteforge archive https://example.com/page --archive-id example-page
siteforge archive --input urls.txt -j 8 --archive-id docs-batch
siteforge archive https://example.com/docs --full-site --max-depth 3 --max-pages 200 -j 4
siteforge archive https://example.com/docs --full-site --progress never
siteforge list
siteforge search "query text" --archive-id docs-batch
siteforge export docs-batch --format all
siteforge pack docs-batch --for-ai --max-tokens 120000

Features

• Polite async HTTP fetching with retries, redirects, compression, timeouts, per-origin pacing, configurable delay, robots.txt crawl-delay checks, and shared per-origin robots lookups for concurrent crawls.
• URL normalization, same-domain/path-prefix crawl scope, include/exclude glob filters, persistent frontier, and resume support.
• Sitemap discovery from robots Sitemap: directives and /sitemap.xml, including sitemap indexes and gzip XML sitemaps.
• Optional JavaScript-rendered DOM capture via a headless browser command such as Chromium/Chrome.
• Optional --render-js-auto mode renders only thin SPA-like pages instead of every page.
• Archive output profiles: full preserves all artifacts, agent skips duplicate plain text/YAML Markdown while keeping raw evidence and agent-readable files, and lean keeps agent-readable Markdown/JSON/chunks while omitting raw payloads and downloaded assets.
• Raw HTML and asset storage with BLAKE3 content hashes, checksum manifests, and total archive size limits.
• HTML extraction for titles, headings, prose, lists, tables, links, code blocks, images, captions, alt text, canonical URLs, metadata, link relations, JSON-LD structured data, downloadable assets, scripts, stylesheets, manifests, media, iframes, embeds, and other page resources.
• Clean Markdown with YAML frontmatter and source attribution, plus basic no-frontmatter Markdown optimized for simple agent ingestion.
• Structured page JSON and JSONL semantic chunks for RAG/AI ingestion.
• AGENTS.md and agent-index.json entrypoints so agents can find pages, assets, chunks, packs, raw payloads, and readable formats quickly.
• .thesa/manifest.json and .thesa/checksums.blake3 sidecars are written for thesa-compatible archive verification.
• SQLite metadata with FTS5 search when available and LIKE fallback otherwise.
• CLI archive and resume runs render a lightweight in-place Aisling tqdm-style progress loader on stderr, with color and a Ctrl+C safe-stop/resume hint, without starting the full TUI. Use --progress auto, --progress always, or --progress never.
• siteforge diagnostics exposes bounded resource visibility for archive root, process RSS/open files where available, archive sizes, crawler counters, job state, and configured limits.
• Long-running views stay bounded: network bodies are streamed up to configured page/asset byte caps, crawl writes use a cached archive-size budget instead of rescanning the archive on every file write, progress stats are refreshed on meaningful phase changes and periodic ticks instead of every hot crawl event, job snapshots reuse one crawl stat aggregate per view refresh, persisted crawl logs keep the newest 5,000 rows, TUI page/asset/job lists load fixed-size windows, readable-file search, WARC export, crawl-time readable/agent index generation, and checksum verification scan metadata in batches/streams, AI pack generation loads chunks only for the oversized page being packed, and zip/checksum generation stream files instead of buffering whole outputs in memory.
• AI context packs split by token budget; oversized pages fall back to semantic JSONL chunks without splitting code chunks.
• Downloaded source assets are included as labeled code blocks/chunks when they are text/code files.
• Image dimensions are probed for downloaded images without fully decoding them.
• Optional OCR backend interface; the default backend shells out to Tesseract if enabled and available.
• Scrin TUI dashboard with tabs, command palette, help/details popups, quit modal, status toasts, job queue, crawl progress, current URL, logs, archive browser, page preview, asset/OCR status, search, and config.

Only pass --header or --cookie for content you are authorized to archive. Siteforge does not bypass authentication, paywalls, CAPTCHAs, DRM, or access controls.

The archive is file-first: readable/README.md, Markdown, JSON, text, JSONL chunks, and packs are the primary AI artifacts. SQLite is kept compact for crawl state and metadata; database-backed search/indexing can become a plugin later.

For agent handoff, use siteforge export <archive_id> --format agent-tar. It writes a compressed <archive_id>-agent.tar.gz containing AGENTS.md, agent-index.json, readable Markdown/basic Markdown/JSON/text, chunks, packs, raw pages, rendered DOM captures, and raw assets while skipping transient export/database side files.

Archive Layout

archives/
  <archive_id>/
    manifest.json
    AGENTS.md
    agent-index.json
    crawl.db
    raw/
      pages/
      rendered/
      assets/
    readable/
      README.md
      AGENTS.md
      basic_markdown/
      markdown/
      json/
      text/
    chunks/
      chunks.jsonl
    packs/
    logs/
    checksums.json
    .thesa/
      manifest.json
      checksums.blake3

checksums.json records deterministic BLAKE3 hashes and byte sizes for archive artifacts. Export outputs and transient SQLite WAL/SHM files are excluded from the checksum manifest. The .thesa/ sidecar uses thesa's manifest/checksum schema and is written after completed archives so thesa verify <archive_path> can validate standalone siteforge archives.

Configuration

Run siteforge config to print the active config and create the platform config file if missing. The default path uses the platform config directory, such as XDG config on Linux.

See config.example.yaml for all supported fields.

Library API

Use Siteforge for programmatic callers such as thesa integrations. The lower-level modules remain public for advanced control, but the facade covers normal archive, resume, search, export, pack, and verify flows. ArchiveRunOptions has constructors and builder methods for cases that need headers, cookies, archive IDs, or crawl limits.

use siteforge::{ArchiveRunOptions, Config, Siteforge};

#[tokio::main]
async fn main() -> siteforge::Result<()> {
    let mut config = Config::default();
    config.archive_root = "archives".into();

    let siteforge = Siteforge::new(config);
    let summary = siteforge.archive_page("https://example.com/").await?;

    println!("archived {} at {}", summary.archive_id, summary.archive_path.display());

    let options = ArchiveRunOptions::full_site_url("https://example.com/docs/", siteforge.config())?
        .with_archive_id("example-docs")
        .with_max_pages(250)
        .with_header("Authorization", "Bearer <token>");
    let docs = siteforge.archive(options).await?;
    println!("thesa sidecar: {}", docs.thesa_manifest_path().display());

    Ok(())
}

Completed archives include .thesa/manifest.json and .thesa/checksums.blake3, so callers can validate the archive with thesa verify <archive_path> without shelling back into siteforge.

TUI Notes

• The TUI uses scrin::Terminal, scrin::EventRouter, Scrin layouts/widgets, and Scrin Markdown rendering.
• Navigation uses Scrin tabs; Ctrl+K/Ctrl+P opens the Scrin command palette, ? opens help, d opens selection details, and q opens a confirmation modal.
• Feedback uses Scrin overlays/popups/toasts so errors and state transitions are visible without digging through logs.
• Loaders are rendered with scrin::effects::LoaderPlayer, which is backed by aisling frames.
• The redraw loop is nonblocking and refreshes archive/job/search state while continuing to process input.
• The dashboard intentionally shows bounded page, asset, job, and log windows with total counts so large archives remain responsive.
• Terminal restore is handled on normal exit and by Scrin’s terminal drop guard during unwinds.

Limitations

• Crawling is intentionally conservative: same-domain and seed path-prefix scoped by default.
• JavaScript-rendered content requires --render-js or --render-js-auto and an installed browser or custom --browser-command. For authorized JS-heavy docs, use --browser-profile <dir> with a browser profile you control, or provide a custom command template that outputs rendered HTML to stdout.
• WARC export is currently WARC-like JSONL metadata pointing at raw payload files, not a standards-complete WARC writer.
• OCR is optional and local only; unavailable OCR records a clear status and does not fail the archive.
• Page and asset response bodies are streamed up to max_page_size_bytes and max_asset_size_bytes; total archive size is enforced with a cached crawl budget before writes.
• Ctrl+C marks active frontier rows pending again, checkpoints the database, and skips expensive index/checksum regeneration so interrupted crawls stop quickly and can be resumed.