# thesa
`thesa` is a Rust CLI to archive GitHub repositories, ML models, and websites.
Repository refresh helpers live in `repoforge`, model target parsing lives in
`modelforge`, and website archiving is powered by `siteforge`.
Made by Trevor Knott for Knott Dynamics. The terminal experience is built with
Scrin and Aisling for the TUI, dashboards, loaders, and visual effects.
`thesa` is the operator-facing binary for the Knott Dynamics archive stack. It
keeps reusable parsing and repository refresh logic in focused crates, then adds
CLI/TUI orchestration, archive manifests, integrity sidecars, and terminal
workflows.
## What It Archives
- GitHub repositories by owner or `owner/repo`.
- Existing Git repository archives via safe refresh/pull workflows.
- Hugging Face, Ollama, and CivitAI model targets.
- Websites and documentation through Siteforge.
## What It Produces
- Local archive directories under `./archives` by default.
- `.thesa/manifest.json` and `.thesa/checksums.blake3` integrity sidecars.
- Scrin/Aisling TUI dashboards for interactive archive flows.
- Siteforge AI-readable website bundles with `AGENTS.md`, indexes, chunks, and raw crawl data.
## Install
`cargo install --path .`
From crates.io after publishing:
`cargo install thesa`
Then run:
`thesa`
Or force the interactive TUI explicitly:
`thesa --tui`
Short form:
`thesa -T`
## Usage
`thesa [TARGET] [options]`
New command form:
- `thesa archive site <URL>`
- `thesa archive url <URL>`
- `thesa archive refresh`
- `thesa archive list`
- `thesa archive inspect <ARCHIVE_ID>`
- `thesa archive search <QUERY>`
- `thesa archive export <ARCHIVE_ID> --format <FORMAT>`
- `thesa archive pack <ARCHIVE_ID>`
- `thesa archive verify <ARCHIVE_ID>`
- `thesa archive resume <ARCHIVE_ID>`
- `thesa archive diagnostics [ARCHIVE_ID]`
- `thesa verify <ARCHIVE_DIR>`
Run with no arguments for interactive mode.
Use `--tui` or `-T` to force interactive mode. If `TARGET` is also passed, the
TUI flag takes precedence and opens the TUI prompt instead.
The TUI landing screen offers three archive routes: GitHub repositories, ML
models, and Siteforge website archives. Selecting Siteforge prompts for a URL
and then shows a preflight options screen for full-site vs single-page archives,
max depth, max pages, archive output profile, and JavaScript rendering mode. TUI
Siteforge runs archive into `./archives/sites` by default.
`TARGET` can be one of: `owner`, `owner/repo`, `https://github.com/owner`,
`https://github.com/owner/repo`.
For model mode, `TARGET` can also be presets:
- `top` (most downloaded)
- `latest` (most recently updated)
For model mode, set `--mode models` and choose one provider:
- `--provider hf` (Hugging Face, default)
- `--provider ollama` (local Ollama instance)
- `--provider civitai` (CivitAI)
In interactive/TUI model mode, provider selection appears before the model target
prompt.
Common options:
- `--depth <DEPTH>`: clone with commit depth
- `-c, --concurrency <N>`: number of repos cloned in parallel (default `4`)
- `--filter <PATTERN>`: filter repository names by substring
- `--skip-existing`: skip repos that are already present
- `--token <TOKEN>`: GitHub API token (or use `GITHUB_TOKEN` env)
- `--hf-token <TOKEN>`: Hugging Face API token (or use `HF_TOKEN` env)
- `--civitai-token <TOKEN>`: CivitAI API token (or use `CIVITAI_TOKEN` env)
- `--hf-mirror`: use `hf-mirror.com` for Hugging Face API calls
- `--provider <PROVIDER>`: model provider for model mode (`hf`, `ollama`, `civitai`)
- `-T, --tui`: force the interactive Scrin interface
Git archive refresh options:
- `thesa archive refresh`: scan `./archives/**` for GitHub worktrees and run `git pull --ff-only`
- `thesa archive update` and `thesa archive pull`: aliases for `refresh`
- `--archive-root <DIR>`: root to scan recursively (default `./archives`)
- `-c, --concurrency <N>`: number of repositories refreshed at once (default `4`)
- `--filter <TEXT>`: refresh only matching owner/repo slugs or paths
- `--dry-run`: print discovered archives without pulling
Refresh only touches actual Git worktrees. It derives `owner/repo` from each
GitHub `origin` URL when available and falls back to the archive path layout,
such as `archives/<owner>/<repo>`. Dirty worktrees are skipped with an error so
local edits are not overwritten silently. The discovery, slugging, clean-check,
and `git pull --ff-only` execution are provided by the `repoforge` crate; thesa
adds archive manifests and terminal reporting around that core.
Website archive options:
- `-o, --output <DIR>`: output root for website archives (default `./archives/sites`)
- `--input <FILE>`: seed URLs from a file, one per line (`#` comments allowed); may be used with or without a positional URL
- `--full-site`: follow in-scope site links instead of archiving one page
- `--allow-cross-domain`: permit cross-domain links
- `--max-depth <N>`: limit crawl depth
- `--max-pages <N>`: limit page count
- `-c, -j, --concurrency <N>`: number of concurrent fetch workers
- `--delay-ms <N>`: politeness delay between requests per worker
- `--archive-id <ID>`: stable archive directory id
- `--include <GLOB>`: include URL glob pattern, repeatable
- `--exclude <GLOB>`: exclude URL glob pattern, repeatable
- `--archive-profile <full|agent|lean>`: choose Siteforge artifact volume
- `--max-page-size-bytes <N>`: maximum bytes fetched per HTML page
- `--max-asset-size-bytes <N>`: maximum bytes fetched per asset
- `--max-total-archive-size-bytes <N>`: maximum bytes written by the archive
- `--render-js`: use Siteforge browser rendering for JavaScript pages
- `--render-js-auto`: render only thin JavaScript app shells
- `--browser-command <COMMAND>`: browser command template for JavaScript rendering
- `--browser-profile <DIR>`: browser profile directory for JavaScript rendering
- `--render-wait-ms <N>`: milliseconds to wait before extracting rendered DOM
- `--header 'Name: Value'`: extra request header, repeatable
- `--cookie <COOKIE_HEADER>`: cookie header value for authenticated/public crawls
- `--user-agent <VALUE>`: Siteforge fetch/render User-Agent
- `--timeout-secs <N>`: HTTP timeout per request
- `--retry-count <N>`: retry attempts for failed frontier items
- `--ocr`: enable Siteforge OCR for code-like downloaded assets
- `--dry-run`: print the Siteforge crawl plan without fetching pages
Avoid storing secrets in docs or shell history when using `--header` or
`--cookie`; prefer a short-lived shell variable or a local wrapper script outside
the repository.
Siteforge archives write an AI-readable bundle under the archive id directory:
- `AGENTS.md`: first file to open for agent-readable instructions/context
- `agent-index.json`: machine-readable index for agents/tools
- `readable/markdown`, `readable/text`, `readable/json`: extracted page content
- `chunks/chunks.jsonl`: chunk stream for downstream indexing
- `manifest.json`, `checksums.json`, `crawl.db`: Siteforge crawl metadata
- `.thesa/manifest.json`, `.thesa/checksums.blake3`: thesa integrity sidecars
Siteforge management commands default to `./archives/sites`. Use
`--archive-root <DIR>` when archives live elsewhere.
Siteforge export formats include `markdown`, `basic-markdown`, `jsonl`, `json`,
`html`, `warc`, `zip`, `tar`, `agent-tar`, and `all`.
## Forge Crates
`thesa` keeps reusable archive logic in small crates so it can stay focused on
CLI/TUI orchestration:
- `repoforge`: discovers Git worktrees under archive roots, derives GitHub
`owner/repo` slugs, rejects dirty worktrees, and refreshes clean clones with
`git pull --ff-only`.
- `modelforge`: normalizes Hugging Face, Ollama, and CivitAI target strings into
provider-neutral target enums before thesa calls provider APIs.
- `siteforge`: captures websites into AI-readable local archives and provides
website archive management commands.
The thesa binary combines those crates with Scrin and Aisling to provide the
interactive command deck, repo/model pickers, clone dashboards, Siteforge option
screens, and archive progress displays.
## Package Safety
The Cargo package is explicitly whitelisted. Releases include only the manifest,
lockfile, README, license, changelog, source, and helper scripts. Archive output,
sessions, notes, logs, build directories, and local scratch files are not shipped.
Included package files:
- `Cargo.toml`
- `Cargo.lock`
- `README.md`
- `LICENSE`
- `CHANGELOG.md`
- `src/**`
- `git.sh`
- `scripts/**`
Repo picker controls (TTY only):
- `Space`: check or uncheck a repo
- Mouse: hover a visible row to focus it, left-click to check or uncheck it
- `a`: check all visible repos
- `c`: clear filter and checks
- `b`: open batch-size overlay (`1`, `2`, `3`, `6`, `8`)
- `PageUp`/`PageDown`: move cursor by one viewport page
- `Home`/`End`: jump to first/last visible result
- `Enter`: clone checked repos only, or visible repos when nothing is checked
Clone dashboard controls (TTY only):
- `t`: toggle repository detail lines in the live batch display
- `q`: request stop after current batch completes
- `m`: motion effect
- `b`: Burn effect
- `w`: Wipe effect
TTY clone runs use a Scrin dashboard. Aisling renders the top Matrix deck,
effect pane, and TQDM-style loader rows for concurrent repository pulls. The TUI
is built against newer Scrin interaction support and remains keyboard-first for
SSH reliability.
Mode, provider, repository, and model picker screens also accept mouse hover and
left-click actions when the terminal supports mouse reporting.
Examples:
- Start TUI and pick target/repo interactively:
- `thesa`
- `thesa --tui`
- `thesa -T`
- Clone one repo:
- `thesa torvalds/linux --output ./repos`
- Clone all public repos for a user/org with shallow history:
- `thesa octocat --output ./repos --depth 1`
- Clone with explicit concurrency:
- `thesa octocat --concurrency 8 --output ./repos`
- Refresh existing GitHub archives:
- `thesa archive refresh --dry-run`
- `thesa archive refresh --archive-root ./archives --concurrency 8`
- `thesa archive pull --filter octocat`
- Archive a website:
- `thesa archive site https://example.com --full-site --max-pages 100`
- `thesa archive url https://example.com/docs --max-depth 2`
- `thesa archive site https://docs.example.com --archive-profile agent --render-js-auto`
- `thesa archive --dry-run url https://example.com/docs --full-site`
- `thesa archive url https://example.com/docs --dry-run --include '**/docs/**' --delay-ms 500`
- `thesa archive url https://example.com/docs --input seeds.txt --archive-id docs`
- `thesa archive url --input seeds.txt --full-site --archive-id docs`
- Manage Siteforge archives:
- `thesa archive list`
- `thesa archive inspect docs`
- `thesa archive search "install" --archive-id docs --limit 5`
- `thesa archive export docs --format agent-tar --output ./exports/docs-agent`
- `thesa archive pack docs --max-tokens 80000`
- `thesa archive verify docs`
- `thesa archive resume docs`
- `thesa archive diagnostics docs`
- Verify a thesa archive directory:
- `thesa verify ./archives/example`
- `thesa verify ./archives/example --json`
- Preview actions before cloning:
- `thesa org-name --dry-run`
- Models and providers:
- `thesa --mode models --provider hf meta-llama --dry-run`
- `thesa --mode models --provider hf meta-llama/Llama-3.1-8B-Instruct --dry-run`
- `thesa --mode models --provider ollama llama3 --dry-run`
- `thesa --mode models --provider hf top --dry-run`
- `thesa --mode models --provider hf latest --dry-run`
- `thesa --mode models --provider ollama top --dry-run`
- `thesa --mode models --provider ollama latest --dry-run`
- `thesa --mode models --provider civitai top --dry-run`
- `thesa --mode models --provider civitai latest --dry-run`
- `thesa --mode models --provider civitai 827184 --dry-run`
## Scripts
You can compile and run with helper scripts:
- Run from repo root (interactive target prompt):
- `./run.sh`
- Pass a target directly:
- `./run.sh <target> [output-root] [concurrency] [extra thesa flags]`
- Compile only:
- `./scripts/compile.sh`
- Compile and run (default concurrency `4`):
- `./scripts/run.sh <target> [output-root] [concurrency] [extra thesa flags excluding --output]`
- Publish release to crates.io:
- `./scripts/publish.sh --allow-dirty`
- `./scripts/publish.sh --dry-run` (safe checks only)
`run.sh` creates an output folder from the target name (for example `owner` or,
for `owner/repo`, just `repo`) and runs clones inside that folder.
## Notes
- `--concurrency` sets the concurrent clone batch size.
- private repositories are intentionally skipped/blocked.
- Cargo packages exclude archive output, session files, logs, target directories,
and temporary test archives. Do not put secrets in project files.