# thesa
`thesa` is a Rust CLI to archive GitHub repositories, ML models, and websites.
Repository refresh helpers live in `repoforge`, model target parsing lives in
`modelforge`, and website archiving is powered by `siteforge`.
Made by Trevor Knott for Knott Dynamics. The terminal experience is built with
Scrin and Aisling for the TUI, dashboards, loaders, and visual effects.
## Install
`cargo install --path .`
From crates.io after publishing:
`cargo install thesa`
Then run:
`thesa`
Or force the interactive TUI explicitly:
`thesa --tui`
Short form:
`thesa -T`
## Usage
`thesa [TARGET] [options]`
New command form:
- `thesa archive site <URL>`
- `thesa archive url <URL>`
- `thesa archive refresh`
- `thesa archive list`
- `thesa archive inspect <ARCHIVE_ID>`
- `thesa archive search <QUERY>`
- `thesa archive export <ARCHIVE_ID> --format <FORMAT>`
- `thesa archive pack <ARCHIVE_ID>`
- `thesa archive verify <ARCHIVE_ID>`
- `thesa archive resume <ARCHIVE_ID>`
- `thesa archive diagnostics [ARCHIVE_ID]`
- `thesa verify <ARCHIVE_DIR>`
Run with no arguments for interactive mode.
Use `--tui` or `-T` to force interactive mode. If `TARGET` is also passed, the
TUI flag takes precedence and opens the TUI prompt instead.
The TUI landing screen offers three archive routes: GitHub repositories, ML
models, and Siteforge website archives. Selecting Siteforge prompts for a URL
and then shows a preflight options screen for full-site vs single-page archives,
max depth, max pages, archive output profile, and JavaScript rendering mode. TUI
Siteforge runs archive into `./archives/sites` by default.
`TARGET` can be one of: `owner`, `owner/repo`, `https://github.com/owner`,
`https://github.com/owner/repo`.
For model mode, `TARGET` can also be presets:
- `top` (most downloaded)
- `latest` (most recently updated)
For model mode, set `--mode models` and choose one provider:
- `--provider hf` (Hugging Face, default)
- `--provider ollama` (local Ollama instance)
- `--provider civitai` (CivitAI)
In interactive/TUI model mode, provider selection appears before the model target
prompt.
Common options:
- `--depth <DEPTH>`: clone with commit depth
- `-c, --concurrency <N>`: number of repos cloned in parallel (default `4`)
- `--filter <PATTERN>`: filter repository names by substring
- `--skip-existing`: skip repos that are already present
- `--token <TOKEN>`: GitHub API token (or use `GITHUB_TOKEN` env)
- `--hf-token <TOKEN>`: Hugging Face API token (or use `HF_TOKEN` env)
- `--civitai-token <TOKEN>`: CivitAI API token (or use `CIVITAI_TOKEN` env)
- `--hf-mirror`: use `hf-mirror.com` for Hugging Face API calls
- `--provider <PROVIDER>`: model provider for model mode (`hf`, `ollama`, `civitai`)
- `-T, --tui`: force the interactive Scrin interface
Git archive refresh options:
- `thesa archive refresh`: scan `./archives/**` for GitHub worktrees and run `git pull --ff-only`
- `thesa archive update` and `thesa archive pull`: aliases for `refresh`
- `--archive-root <DIR>`: root to scan recursively (default `./archives`)
- `-c, --concurrency <N>`: number of repositories refreshed at once (default `4`)
- `--filter <TEXT>`: refresh only matching owner/repo slugs or paths
- `--dry-run`: print discovered archives without pulling
Refresh only touches actual Git worktrees. It derives `owner/repo` from each
GitHub `origin` URL when available and falls back to the archive path layout,
such as `archives/<owner>/<repo>`. Dirty worktrees are skipped with an error so
local edits are not overwritten silently. The discovery, slugging, clean-check,
and `git pull --ff-only` execution are provided by the `repoforge` crate; thesa
adds archive manifests and terminal reporting around that core.
Website archive options:
- `-o, --output <DIR>`: output root for website archives (default `./archives/sites`)
- `--input <FILE>`: seed URLs from a file, one per line (`#` comments allowed); may be used with or without a positional URL
- `--full-site`: follow in-scope site links instead of archiving one page
- `--allow-cross-domain`: permit cross-domain links
- `--max-depth <N>`: limit crawl depth
- `--max-pages <N>`: limit page count
- `-c, -j, --concurrency <N>`: number of concurrent fetch workers
- `--delay-ms <N>`: politeness delay between requests per worker
- `--archive-id <ID>`: stable archive directory id
- `--include <GLOB>`: include URL glob pattern, repeatable
- `--exclude <GLOB>`: exclude URL glob pattern, repeatable
- `--archive-profile <full|agent|lean>`: choose Siteforge artifact volume
- `--max-page-size-bytes <N>`: maximum bytes fetched per HTML page
- `--max-asset-size-bytes <N>`: maximum bytes fetched per asset
- `--max-total-archive-size-bytes <N>`: maximum bytes written by the archive
- `--render-js`: use Siteforge browser rendering for JavaScript pages
- `--render-js-auto`: render only thin JavaScript app shells
- `--browser-command <COMMAND>`: browser command template for JavaScript rendering
- `--browser-profile <DIR>`: browser profile directory for JavaScript rendering
- `--render-wait-ms <N>`: milliseconds to wait before extracting rendered DOM
- `--header 'Name: Value'`: extra request header, repeatable
- `--cookie <COOKIE_HEADER>`: cookie header value for authenticated/public crawls
- `--user-agent <VALUE>`: Siteforge fetch/render User-Agent
- `--timeout-secs <N>`: HTTP timeout per request
- `--retry-count <N>`: retry attempts for failed frontier items
- `--ocr`: enable Siteforge OCR for code-like downloaded assets
- `--dry-run`: print the Siteforge crawl plan without fetching pages
Avoid storing secrets in docs or shell history when using `--header` or
`--cookie`; prefer a short-lived shell variable or a local wrapper script outside
the repository.
Siteforge archives write an AI-readable bundle under the archive id directory:
- `AGENTS.md`: first file to open for agent-readable instructions/context
- `agent-index.json`: machine-readable index for agents/tools
- `readable/markdown`, `readable/text`, `readable/json`: extracted page content
- `chunks/chunks.jsonl`: chunk stream for downstream indexing
- `manifest.json`, `checksums.json`, `crawl.db`: Siteforge crawl metadata
- `.thesa/manifest.json`, `.thesa/checksums.blake3`: thesa integrity sidecars
Siteforge management commands default to `./archives/sites`. Use
`--archive-root <DIR>` when archives live elsewhere.
Siteforge export formats include `markdown`, `basic-markdown`, `jsonl`, `json`,
`html`, `warc`, `zip`, `tar`, `agent-tar`, and `all`.
## Forge Crates
`thesa` keeps reusable archive logic in small crates so it can stay focused on
CLI/TUI orchestration:
- `repoforge`: discovers Git worktrees under archive roots, derives GitHub
`owner/repo` slugs, rejects dirty worktrees, and refreshes clean clones with
`git pull --ff-only`.
- `modelforge`: normalizes Hugging Face, Ollama, and CivitAI target strings into
provider-neutral target enums before thesa calls provider APIs.
- `siteforge`: captures websites into AI-readable local archives and provides
website archive management commands.
The thesa binary combines those crates with Scrin and Aisling to provide the
interactive command deck, repo/model pickers, clone dashboards, Siteforge option
screens, and archive progress displays.
Repo picker controls (TTY only):
- `Space`: check or uncheck a repo
- Mouse: hover a visible row to focus it, left-click to check or uncheck it
- `a`: check all visible repos
- `c`: clear filter and checks
- `b`: open batch-size overlay (`1`, `2`, `3`, `6`, `8`)
- `PageUp`/`PageDown`: move cursor by one viewport page
- `Home`/`End`: jump to first/last visible result
- `Enter`: clone checked repos only, or visible repos when nothing is checked
Clone dashboard controls (TTY only):
- `t`: toggle repository detail lines in the live batch display
- `q`: request stop after current batch completes
- `m`: motion effect
- `b`: Burn effect
- `w`: Wipe effect
TTY clone runs use a Scrin dashboard. Aisling renders the top Matrix deck,
effect pane, and TQDM-style loader rows for concurrent repository pulls. The TUI
is built against newer Scrin interaction support and remains keyboard-first for
SSH reliability.
Mode, provider, repository, and model picker screens also accept mouse hover and
left-click actions when the terminal supports mouse reporting.
Examples:
- Start TUI and pick target/repo interactively:
- `thesa`
- `thesa --tui`
- `thesa -T`
- Clone one repo:
- `thesa torvalds/linux --output ./repos`
- Clone all public repos for a user/org with shallow history:
- `thesa octocat --output ./repos --depth 1`
- Clone with explicit concurrency:
- `thesa octocat --concurrency 8 --output ./repos`
- Refresh existing GitHub archives:
- `thesa archive refresh --dry-run`
- `thesa archive refresh --archive-root ./archives --concurrency 8`
- `thesa archive pull --filter octocat`
- Archive a website:
- `thesa archive site https://example.com --full-site --max-pages 100`
- `thesa archive url https://example.com/docs --max-depth 2`
- `thesa archive site https://docs.example.com --archive-profile agent --render-js-auto`
- `thesa archive --dry-run url https://example.com/docs --full-site`
- `thesa archive url https://example.com/docs --dry-run --include '**/docs/**' --delay-ms 500`
- `thesa archive url https://example.com/docs --input seeds.txt --archive-id docs`
- `thesa archive url --input seeds.txt --full-site --archive-id docs`
- Manage Siteforge archives:
- `thesa archive list`
- `thesa archive inspect docs`
- `thesa archive search "install" --archive-id docs --limit 5`
- `thesa archive export docs --format agent-tar --output ./exports/docs-agent`
- `thesa archive pack docs --max-tokens 80000`
- `thesa archive verify docs`
- `thesa archive resume docs`
- `thesa archive diagnostics docs`
- Verify a thesa archive directory:
- `thesa verify ./archives/example`
- `thesa verify ./archives/example --json`
- Preview actions before cloning:
- `thesa org-name --dry-run`
- Models and providers:
- `thesa --mode models --provider hf meta-llama --dry-run`
- `thesa --mode models --provider hf meta-llama/Llama-3.1-8B-Instruct --dry-run`
- `thesa --mode models --provider ollama llama3 --dry-run`
- `thesa --mode models --provider hf top --dry-run`
- `thesa --mode models --provider hf latest --dry-run`
- `thesa --mode models --provider ollama top --dry-run`
- `thesa --mode models --provider ollama latest --dry-run`
- `thesa --mode models --provider civitai top --dry-run`
- `thesa --mode models --provider civitai latest --dry-run`
- `thesa --mode models --provider civitai 827184 --dry-run`
## Scripts
You can compile and run with helper scripts:
- Run from repo root (interactive target prompt):
- `./run.sh`
- Pass a target directly:
- `./run.sh <target> [output-root] [concurrency] [extra thesa flags]`
- Compile only:
- `./scripts/compile.sh`
- Compile and run (default concurrency `4`):
- `./scripts/run.sh <target> [output-root] [concurrency] [extra thesa flags excluding --output]`
- Publish release to crates.io:
- `./scripts/publish.sh --allow-dirty`
- `./scripts/publish.sh --dry-run` (safe checks only)
`run.sh` creates an output folder from the target name (for example `owner` or,
for `owner/repo`, just `repo`) and runs clones inside that folder.
## Notes
- `--concurrency` sets the concurrent clone batch size.
- private repositories are intentionally skipped/blocked.
- Cargo packages exclude archive output, session files, logs, target directories,
and temporary test archives. Do not put secrets in project files.