siteforge 0.1.1

Archive websites into AI-readable local knowledge archives
Documentation

siteforge

siteforge is a Rust CLI/TUI tool for archiving websites into local, AI-readable knowledge archives. It stores raw snapshots, structured metadata, cleaned Markdown, JSON, JSONL semantic chunks, searchable SQLite metadata, and token-budgeted AI context packs.

The TUI is built with scrin as a hard dependency. Terminal loaders and status effects use aisling through Scrin’s Aisling-backed LoaderPlayer, with direct aisling::LoaderProgress progress data in the dashboard.

Examples

siteforge archive https://catlikecoding.com/unity/tutorials/ --full-site
siteforge archive https://example.com/article
siteforge archive --input urls.txt -j 8
siteforge archive https://catlikecoding.com/unity/tutorials/ --full-site --concurrency 4 --delay-ms 1000
siteforge archive https://docs.example.test/private --header "Authorization: Bearer <token>" --cookie "session=..."
siteforge archive https://app.example.test/docs --render-js --browser-command chromium --browser-profile ~/.config/chromium-siteforge
siteforge tui
siteforge list
siteforge inspect catlikecoding-20260629120000
siteforge search "mesh deformation code"
siteforge export catlikecoding-20260629120000 --format markdown
siteforge export catlikecoding-20260629120000 --format basic-markdown
siteforge export catlikecoding-20260629120000 --format agent-tar
siteforge export catlikecoding-20260629120000 --format zip
siteforge export catlikecoding-20260629120000 --format tar
siteforge pack catlikecoding-20260629120000 --for-ai --max-tokens 120000
siteforge verify catlikecoding-20260629120000
siteforge resume catlikecoding-20260629120000
siteforge config

Build And Test

cargo fmt
cargo check
cargo clippy
cargo test
cargo run -- archive https://example.com/article
cargo run -- tui

Basic CLI

Use -j/--concurrency/--jobs to control how many page fetches can run concurrently. Per-origin politeness still applies through configured delay, robots crawl-delay, and robots request-rate.

siteforge archive https://example.com/page --archive-id example-page
siteforge archive --input urls.txt -j 8 --archive-id docs-batch
siteforge archive https://example.com/docs --full-site --max-depth 3 --max-pages 200 -j 4
siteforge list
siteforge search "query text" --archive-id docs-batch
siteforge export docs-batch --format all
siteforge pack docs-batch --for-ai --max-tokens 120000

Features

  • Polite async HTTP fetching with retries, redirects, compression, timeouts, per-origin pacing, configurable delay, and robots.txt crawl-delay checks.
  • URL normalization, same-domain/path-prefix crawl scope, include/exclude glob filters, persistent frontier, and resume support.
  • Sitemap discovery from robots Sitemap: directives and /sitemap.xml, including sitemap indexes and gzip XML sitemaps.
  • Optional JavaScript-rendered DOM capture via a headless browser command such as Chromium/Chrome.
  • Raw HTML and asset storage with BLAKE3 content hashes, checksum manifests, and total archive size limits.
  • HTML extraction for titles, headings, prose, lists, tables, links, code blocks, images, captions, alt text, canonical URLs, metadata, link relations, JSON-LD structured data, downloadable assets, scripts, stylesheets, manifests, media, iframes, embeds, and other page resources.
  • Clean Markdown with YAML frontmatter and source attribution, plus basic no-frontmatter Markdown optimized for simple agent ingestion.
  • Structured page JSON and JSONL semantic chunks for RAG/AI ingestion.
  • AGENTS.md and agent-index.json entrypoints so agents can find pages, assets, chunks, packs, raw payloads, and readable formats quickly.
  • SQLite metadata with FTS5 search when available and LIKE fallback otherwise.
  • CLI archive and resume runs render lightweight Aisling tqdm-style progress loaders on stderr without starting the full TUI.
  • AI context packs split by token budget; oversized pages fall back to semantic JSONL chunks without splitting code chunks.
  • Downloaded source assets are included as labeled code blocks/chunks when they are text/code files.
  • Image dimensions are probed for downloaded images without fully decoding them.
  • Optional OCR backend interface; the default backend shells out to Tesseract if enabled and available.
  • Scrin TUI dashboard with tabs, command palette, help/details popups, quit modal, status toasts, job queue, crawl progress, current URL, logs, archive browser, page preview, asset/OCR status, search, and config.

Only pass --header or --cookie for content you are authorized to archive. Siteforge does not bypass authentication, paywalls, CAPTCHAs, DRM, or access controls.

The archive is file-first: readable/README.md, Markdown, JSON, text, JSONL chunks, and packs are the primary AI artifacts. SQLite is kept compact for crawl state and metadata; database-backed search/indexing can become a plugin later.

For agent handoff, use siteforge export <archive_id> --format agent-tar. It writes a compressed <archive_id>-agent.tar.gz containing AGENTS.md, agent-index.json, readable Markdown/basic Markdown/JSON/text, chunks, packs, raw pages, rendered DOM captures, and raw assets while skipping transient export/database side files.

Archive Layout

archives/
  <archive_id>/
    manifest.json
    AGENTS.md
    agent-index.json
    crawl.db
    raw/
      pages/
      rendered/
      assets/
    readable/
      README.md
      AGENTS.md
      basic_markdown/
      markdown/
      json/
      text/
    chunks/
      chunks.jsonl
    packs/
    logs/
    checksums.json

checksums.json records deterministic BLAKE3 hashes and byte sizes for archive artifacts. Export outputs and transient SQLite WAL/SHM files are excluded from the checksum manifest.

Configuration

Run siteforge config to print the active config and create the platform config file if missing. The default path uses the platform config directory, such as XDG config on Linux.

See config.example.yaml for all supported fields.

TUI Notes

  • The TUI uses scrin::Terminal, scrin::EventRouter, Scrin layouts/widgets, and Scrin Markdown rendering.
  • Navigation uses Scrin tabs; Ctrl+K/Ctrl+P opens the Scrin command palette, ? opens help, d opens selection details, and q opens a confirmation modal.
  • Feedback uses Scrin overlays/popups/toasts so errors and state transitions are visible without digging through logs.
  • Loaders are rendered with scrin::effects::LoaderPlayer, which is backed by aisling frames.
  • The redraw loop is nonblocking and refreshes archive/job/search state while continuing to process input.
  • Terminal restore is handled on normal exit and by Scrin’s terminal drop guard during unwinds.

Limitations

  • Crawling is intentionally conservative: same-domain and seed path-prefix scoped by default.
  • JavaScript-rendered content requires --render-js and an installed browser or custom --browser-command. For authorized JS-heavy docs, use --browser-profile <dir> with a browser profile you control, or provide a custom command template that outputs rendered HTML to stdout.
  • WARC export is currently WARC-like JSONL metadata pointing at raw payload files, not a standards-complete WARC writer.
  • OCR is optional and local only; unavailable OCR records a clear status and does not fail the archive.
  • Total archive size is enforced before writes; in-flight network downloads can still temporarily exist in memory before being rejected.