# siteforge
`siteforge` is a Rust CLI/TUI tool for archiving websites into local, AI-readable knowledge archives. It stores raw snapshots, structured metadata, cleaned Markdown, JSON, JSONL semantic chunks, searchable SQLite metadata, and token-budgeted AI context packs.
The TUI is built with `scrin` as a hard dependency. Terminal loaders and status effects use `aisling` through Scrin’s Aisling-backed `LoaderPlayer`, with direct `aisling::LoaderProgress` progress data in the dashboard.
## Examples
```bash
siteforge archive https://catlikecoding.com/unity/tutorials/ --full-site
siteforge archive https://example.com/article
siteforge archive --input urls.txt -j 8
siteforge archive https://catlikecoding.com/unity/tutorials/ --full-site --concurrency 4 --delay-ms 1000
siteforge archive https://docs.example.test/private --header "Authorization: Bearer <token>" --cookie "session=..."
siteforge archive https://app.example.test/docs --render-js --browser-command chromium --browser-profile ~/.config/chromium-siteforge
siteforge archive https://example.com/docs --full-site --progress always
siteforge tui
siteforge list
siteforge inspect catlikecoding-20260629120000
siteforge diagnostics --archive-id catlikecoding-20260629120000
siteforge search "mesh deformation code"
siteforge export catlikecoding-20260629120000 --format markdown
siteforge export catlikecoding-20260629120000 --format basic-markdown
siteforge export catlikecoding-20260629120000 --format agent-tar
siteforge export catlikecoding-20260629120000 --format zip
siteforge export catlikecoding-20260629120000 --format tar
siteforge pack catlikecoding-20260629120000 --for-ai --max-tokens 120000
siteforge verify catlikecoding-20260629120000
siteforge resume catlikecoding-20260629120000
siteforge config
```
## Build And Test
```bash
cargo fmt
cargo check
cargo clippy
cargo test
cargo run -- archive https://example.com/article
cargo run -- tui
```
## Basic CLI
Use `-j`/`--concurrency`/`--jobs` to control how many page fetches can run concurrently. Per-origin politeness still applies through configured delay, robots crawl-delay, and robots request-rate.
```bash
siteforge archive https://example.com/page --archive-id example-page
siteforge archive --input urls.txt -j 8 --archive-id docs-batch
siteforge archive https://example.com/docs --full-site --max-depth 3 --max-pages 200 -j 4
siteforge archive https://example.com/docs --full-site --progress never
siteforge list
siteforge search "query text" --archive-id docs-batch
siteforge export docs-batch --format all
siteforge pack docs-batch --for-ai --max-tokens 120000
```
## Features
- Polite async HTTP fetching with retries, redirects, compression, timeouts, per-origin pacing, configurable delay, and robots.txt crawl-delay checks.
- URL normalization, same-domain/path-prefix crawl scope, include/exclude glob filters, persistent frontier, and resume support.
- Sitemap discovery from robots `Sitemap:` directives and `/sitemap.xml`, including sitemap indexes and gzip XML sitemaps.
- Optional JavaScript-rendered DOM capture via a headless browser command such as Chromium/Chrome.
- Raw HTML and asset storage with BLAKE3 content hashes, checksum manifests, and total archive size limits.
- HTML extraction for titles, headings, prose, lists, tables, links, code blocks, images, captions, alt text, canonical URLs, metadata, link relations, JSON-LD structured data, downloadable assets, scripts, stylesheets, manifests, media, iframes, embeds, and other page resources.
- Clean Markdown with YAML frontmatter and source attribution, plus basic no-frontmatter Markdown optimized for simple agent ingestion.
- Structured page JSON and JSONL semantic chunks for RAG/AI ingestion.
- `AGENTS.md` and `agent-index.json` entrypoints so agents can find pages, assets, chunks, packs, raw payloads, and readable formats quickly.
- SQLite metadata with FTS5 search when available and LIKE fallback otherwise.
- CLI archive and resume runs render a lightweight in-place Aisling tqdm-style progress loader on stderr, with color and a Ctrl+C safe-stop/resume hint, without starting the full TUI. Use `--progress auto`, `--progress always`, or `--progress never`.
- `siteforge diagnostics` exposes bounded resource visibility for archive root, archive sizes, crawler counters, job state, and configured limits.
- AI context packs split by token budget; oversized pages fall back to semantic JSONL chunks without splitting code chunks.
- Downloaded source assets are included as labeled code blocks/chunks when they are text/code files.
- Image dimensions are probed for downloaded images without fully decoding them.
- Optional OCR backend interface; the default backend shells out to Tesseract if enabled and available.
- Scrin TUI dashboard with tabs, command palette, help/details popups, quit modal, status toasts, job queue, crawl progress, current URL, logs, archive browser, page preview, asset/OCR status, search, and config.
Only pass `--header` or `--cookie` for content you are authorized to archive. Siteforge does not bypass authentication, paywalls, CAPTCHAs, DRM, or access controls.
The archive is file-first: `readable/README.md`, Markdown, JSON, text, JSONL chunks, and packs are the primary AI artifacts. SQLite is kept compact for crawl state and metadata; database-backed search/indexing can become a plugin later.
For agent handoff, use `siteforge export <archive_id> --format agent-tar`. It writes a compressed `<archive_id>-agent.tar.gz` containing `AGENTS.md`, `agent-index.json`, readable Markdown/basic Markdown/JSON/text, chunks, packs, raw pages, rendered DOM captures, and raw assets while skipping transient export/database side files.
## Archive Layout
```text
archives/
<archive_id>/
manifest.json
AGENTS.md
agent-index.json
crawl.db
raw/
pages/
rendered/
assets/
readable/
README.md
AGENTS.md
basic_markdown/
markdown/
json/
text/
chunks/
chunks.jsonl
packs/
logs/
checksums.json
```
`checksums.json` records deterministic BLAKE3 hashes and byte sizes for archive artifacts. Export outputs and transient SQLite WAL/SHM files are excluded from the checksum manifest.
## Configuration
Run `siteforge config` to print the active config and create the platform config file if missing. The default path uses the platform config directory, such as XDG config on Linux.
See `config.example.yaml` for all supported fields.
## TUI Notes
- The TUI uses `scrin::Terminal`, `scrin::EventRouter`, Scrin layouts/widgets, and Scrin Markdown rendering.
- Navigation uses Scrin tabs; `Ctrl+K`/`Ctrl+P` opens the Scrin command palette, `?` opens help, `d` opens selection details, and `q` opens a confirmation modal.
- Feedback uses Scrin overlays/popups/toasts so errors and state transitions are visible without digging through logs.
- Loaders are rendered with `scrin::effects::LoaderPlayer`, which is backed by `aisling` frames.
- The redraw loop is nonblocking and refreshes archive/job/search state while continuing to process input.
- Terminal restore is handled on normal exit and by Scrin’s terminal drop guard during unwinds.
## Limitations
- Crawling is intentionally conservative: same-domain and seed path-prefix scoped by default.
- JavaScript-rendered content requires `--render-js` and an installed browser or custom `--browser-command`. For authorized JS-heavy docs, use `--browser-profile <dir>` with a browser profile you control, or provide a custom command template that outputs rendered HTML to stdout.
- WARC export is currently WARC-like JSONL metadata pointing at raw payload files, not a standards-complete WARC writer.
- OCR is optional and local only; unavailable OCR records a clear status and does not fail the archive.
- Total archive size is enforced before writes; in-flight network downloads can still temporarily exist in memory before being rejected.