# siteforge
`siteforge` is a Rust CLI/TUI tool for archiving websites into local, AI-readable knowledge archives. It stores raw snapshots, structured metadata, cleaned Markdown, JSON, JSONL semantic chunks, searchable SQLite metadata, and token-budgeted AI context packs.
The TUI is built with `scrin` as a hard dependency. Terminal loaders and status effects use `aisling` through Scrin’s Aisling-backed `LoaderPlayer`, with direct `aisling::LoaderProgress` progress data in the dashboard.
## Examples
```bash
siteforge archive https://catlikecoding.com/unity/tutorials/ --full-site
siteforge archive https://example.com/article
siteforge archive --input urls.txt -j 8
siteforge archive https://catlikecoding.com/unity/tutorials/ --full-site --concurrency 4 --delay-ms 1000
siteforge archive https://docs.example.test --full-site --archive-profile agent
siteforge archive https://docs.example.test --full-site --archive-profile lean
siteforge archive https://docs.example.test/private --header "Authorization: Bearer <token>" --cookie "session=..."
siteforge archive https://app.example.test/docs --render-js --browser-command chromium --browser-profile ~/.config/chromium-siteforge
siteforge archive https://app.example.test/docs --render-js-auto --browser-command chromium
siteforge archive https://example.com/docs --full-site --progress always
siteforge tui
siteforge list
siteforge inspect catlikecoding-20260629120000
siteforge diagnostics --archive-id catlikecoding-20260629120000
siteforge search "mesh deformation code"
siteforge export catlikecoding-20260629120000 --format markdown
siteforge export catlikecoding-20260629120000 --format basic-markdown
siteforge export catlikecoding-20260629120000 --format agent-tar
siteforge export catlikecoding-20260629120000 --format zip
siteforge export catlikecoding-20260629120000 --format tar
siteforge pack catlikecoding-20260629120000 --for-ai --max-tokens 120000
siteforge verify catlikecoding-20260629120000
siteforge resume catlikecoding-20260629120000
siteforge config
```
## Build And Test
```bash
cargo install siteforge
cargo fmt
cargo check
cargo clippy
cargo test
cargo run -- archive https://example.com/article
cargo run -- tui
```
The published Cargo package excludes local archive outputs, session files, logs, `.opencode/`, and build artifacts.
## Basic CLI
Use `-j`/`--concurrency`/`--jobs` to control how many page fetches can run concurrently. Defaults are tuned for a modest crawl pace, and per-origin politeness still applies through configured delay, robots crawl-delay, and robots request-rate.
```bash
siteforge archive https://example.com/page --archive-id example-page
siteforge archive --input urls.txt -j 8 --archive-id docs-batch
siteforge archive https://example.com/docs --full-site --max-depth 3 --max-pages 200 -j 4
siteforge archive https://example.com/docs --full-site --progress never
siteforge list
siteforge search "query text" --archive-id docs-batch
siteforge export docs-batch --format all
siteforge pack docs-batch --for-ai --max-tokens 120000
```
## Features
- Polite async HTTP fetching with retries, redirects, compression, timeouts, per-origin pacing, configurable delay, robots.txt crawl-delay checks, and shared per-origin robots lookups for concurrent crawls.
- URL normalization, same-domain/path-prefix crawl scope, include/exclude glob filters, persistent frontier, and resume support.
- Sitemap discovery from robots `Sitemap:` directives and `/sitemap.xml`, including sitemap indexes and gzip XML sitemaps.
- Optional JavaScript-rendered DOM capture via a headless browser command such as Chromium/Chrome.
- Optional `--render-js-auto` mode renders only thin SPA-like pages instead of every page.
- Archive output profiles: `full` preserves all artifacts, `agent` skips duplicate plain text/YAML Markdown while keeping raw evidence and agent-readable files, and `lean` keeps agent-readable Markdown/JSON/chunks while omitting raw payloads and downloaded assets.
- Raw HTML and asset storage with BLAKE3 content hashes, checksum manifests, and total archive size limits.
- HTML extraction for titles, headings, prose, lists, tables, links, code blocks, images, captions, alt text, canonical URLs, metadata, link relations, JSON-LD structured data, downloadable assets, scripts, stylesheets, manifests, media, iframes, embeds, and other page resources.
- Clean Markdown with YAML frontmatter and source attribution, plus basic no-frontmatter Markdown optimized for simple agent ingestion.
- Structured page JSON and JSONL semantic chunks for RAG/AI ingestion.
- `AGENTS.md` and `agent-index.json` entrypoints so agents can find pages, assets, chunks, packs, raw payloads, and readable formats quickly.
- `.thesa/manifest.json` and `.thesa/checksums.blake3` sidecars are written for thesa-compatible archive verification.
- SQLite metadata with FTS5 search when available and LIKE fallback otherwise.
- CLI archive and resume runs render a lightweight in-place Aisling tqdm-style progress loader on stderr, with color and a Ctrl+C safe-stop/resume hint, without starting the full TUI. Use `--progress auto`, `--progress always`, or `--progress never`.
- `siteforge diagnostics` exposes bounded resource visibility for archive root, process RSS/open files where available, archive sizes, crawler counters, job state, and configured limits.
- Long-running views stay bounded: network bodies are streamed up to configured page/asset byte caps and preallocated from safe `Content-Length` hints, crawl writes use a cached archive-size budget instead of rescanning the archive on every file write, frontier queue lookups are indexed and batch updates reuse SQLite statements, progress stats are refreshed on meaningful phase changes and periodic ticks instead of every hot crawl event, SQLite FTS availability is cached for indexing/search, job snapshots reuse one crawl stat aggregate per view refresh, persisted crawl logs keep the newest 5,000 rows, TUI page/asset/job lists load fixed-size windows, readable-file search, WARC export, crawl-time readable/agent index generation, and checksum verification scan metadata in batches/streams, AI pack generation loads chunks only for the oversized page being packed, and zip/checksum generation stream files instead of buffering whole outputs in memory.
- AI context packs split by token budget; oversized pages fall back to semantic JSONL chunks without splitting code chunks.
- Downloaded source assets are included as labeled code blocks/chunks when they are text/code files.
- Image dimensions are probed for downloaded images without fully decoding them.
- Optional OCR backend interface; the default backend shells out to Tesseract if enabled and available.
- Scrin TUI dashboard with tabs, command palette, help/details popups, quit modal, status toasts, job queue, crawl progress, current URL, logs, archive browser, page preview, asset/OCR status, search, and config.
Only pass `--header` or `--cookie` for content you are authorized to archive. Siteforge does not bypass authentication, paywalls, CAPTCHAs, DRM, or access controls.
The archive is file-first: `readable/README.md`, Markdown, JSON, text, JSONL chunks, and packs are the primary AI artifacts. SQLite is kept compact for crawl state and metadata; database-backed search/indexing can become a plugin later.
For agent handoff, use `siteforge export <archive_id> --format agent-tar`. It writes a compressed `<archive_id>-agent.tar.gz` containing `AGENTS.md`, `agent-index.json`, readable Markdown/basic Markdown/JSON/text, chunks, packs, raw pages, rendered DOM captures, and raw assets while skipping transient export/database side files.
## Archive Layout
```text
archives/
<archive_id>/
manifest.json
AGENTS.md
agent-index.json
crawl.db
raw/
pages/
rendered/
assets/
readable/
README.md
AGENTS.md
basic_markdown/
markdown/
json/
text/
chunks/
chunks.jsonl
packs/
logs/
checksums.json
.thesa/
manifest.json
checksums.blake3
```
`checksums.json` records deterministic BLAKE3 hashes and byte sizes for archive artifacts. Export outputs and transient SQLite WAL/SHM files are excluded from the checksum manifest.
The `.thesa/` sidecar uses thesa's manifest/checksum schema and is written after completed archives so `thesa verify <archive_path>` can validate standalone siteforge archives.
## Configuration
Run `siteforge config` to print the active config and create the platform config file if missing. The default path uses the platform config directory, such as XDG config on Linux.
See `config.example.yaml` for all supported fields.
## Library API
Use `Siteforge` for programmatic callers such as thesa integrations. The lower-level modules remain public for advanced control, but the facade covers normal archive, resume, search, export, pack, and verify flows. `ArchiveRunOptions` has constructors and builder methods for cases that need headers, cookies, archive IDs, or crawl limits.
```rust
use siteforge::{ArchiveRunOptions, Config, Siteforge};
#[tokio::main]
async fn main() -> siteforge::Result<()> {
let mut config = Config::default();
config.archive_root = "archives".into();
let siteforge = Siteforge::new(config);
let summary = siteforge.archive_page("https://example.com/").await?;
println!("archived {} at {}", summary.archive_id, summary.archive_path.display());
let options = ArchiveRunOptions::full_site_url("https://example.com/docs/", siteforge.config())?
.with_archive_id("example-docs")
.with_max_pages(250)
.with_header("Authorization", "Bearer <token>");
let docs = siteforge.archive(options).await?;
println!("thesa sidecar: {}", docs.thesa_manifest_path().display());
Ok(())
}
```
Completed archives include `.thesa/manifest.json` and `.thesa/checksums.blake3`, so callers can validate the archive with `thesa verify <archive_path>` without shelling back into siteforge.
## TUI Notes
- The TUI uses `scrin::Terminal`, `scrin::EventRouter`, Scrin layouts/widgets, and Scrin Markdown rendering.
- Navigation uses Scrin tabs; `Ctrl+K`/`Ctrl+P` opens the Scrin command palette, `?` opens help, `d` opens selection details, and `q` opens a confirmation modal.
- Feedback uses Scrin overlays/popups/toasts so errors and state transitions are visible without digging through logs.
- Loaders are rendered with `scrin::effects::LoaderPlayer`, which is backed by `aisling` frames.
- The redraw loop is nonblocking and refreshes archive/job/search state while continuing to process input.
- The dashboard intentionally shows bounded page, asset, job, and log windows with total counts so large archives remain responsive.
- Terminal restore is handled on normal exit and by Scrin’s terminal drop guard during unwinds.
## Limitations
- Crawling is intentionally conservative: same-domain and seed path-prefix scoped by default.
- JavaScript-rendered content requires `--render-js` or `--render-js-auto` and an installed browser or custom `--browser-command`. For authorized JS-heavy docs, use `--browser-profile <dir>` with a browser profile you control, or provide a custom command template that outputs rendered HTML to stdout.
- WARC export is currently WARC-like JSONL metadata pointing at raw payload files, not a standards-complete WARC writer.
- OCR is optional and local only; unavailable OCR records a clear status and does not fail the archive.
- Page and asset response bodies are streamed up to `max_page_size_bytes` and `max_asset_size_bytes`; total archive size is enforced with a cached crawl budget before writes.
- Ctrl+C marks active frontier rows pending again, checkpoints the database, and skips expensive index/checksum regeneration so interrupted crawls stop quickly and can be resumed.