# thesa
`thesa` is a Rust CLI to archive GitHub repositories, ML models, and websites.
Website archiving is powered by `siteforge`.
## Install
`cargo install --path .`
From crates.io after publishing:
`cargo install thesa`
Then run:
`thesa`
Or force the interactive TUI explicitly:
`thesa --tui`
Short form:
`thesa -T`
## Usage
`thesa [TARGET] [options]`
New command form:
- `thesa archive site <URL>`
- `thesa archive url <URL>`
- `thesa verify <ARCHIVE_DIR>`
Run with no arguments for interactive mode.
Use `--tui` or `-T` to force interactive mode. If `TARGET` is also passed, the
TUI flag takes precedence and opens the TUI prompt instead.
The TUI landing screen offers three archive routes: GitHub repositories, ML
models, and Siteforge website archives. Selecting Siteforge prompts for a URL
and runs a same-domain full-site crawl into `./archives/sites` by default.
`TARGET` can be one of: `owner`, `owner/repo`, `https://github.com/owner`,
`https://github.com/owner/repo`.
For model mode, `TARGET` can also be presets:
- `top` (most downloaded)
- `latest` (most recently updated)
For model mode, set `--mode models` and choose one provider:
- `--provider hf` (Hugging Face, default)
- `--provider ollama` (local Ollama instance)
- `--provider civitai` (CivitAI)
In interactive/TUI model mode, provider selection appears before the model target
prompt.
Common options:
- `--depth <DEPTH>`: clone with commit depth
- `-c, --concurrency <N>`: number of repos cloned in parallel (default `4`)
- `--filter <PATTERN>`: filter repository names by substring
- `--skip-existing`: skip repos that are already present
- `--token <TOKEN>`: GitHub API token (or use `GITHUB_TOKEN` env)
- `--hf-token <TOKEN>`: Hugging Face API token (or use `HF_TOKEN` env)
- `--civitai-token <TOKEN>`: CivitAI API token (or use `CIVITAI_TOKEN` env)
- `--hf-mirror`: use `hf-mirror.com` for Hugging Face API calls
- `--provider <PROVIDER>`: model provider for model mode (`hf`, `ollama`, `civitai`)
- `-T, --tui`: force the interactive Scrin interface
Website archive options:
- `--full-site`: follow in-scope site links instead of archiving one page
- `--allow-cross-domain`: permit cross-domain links
- `--max-depth <N>`: limit crawl depth
- `--max-pages <N>`: limit page count
- `--include <GLOB>`: include URL glob pattern, repeatable
- `--exclude <GLOB>`: exclude URL glob pattern, repeatable
- `--render-js`: use Siteforge browser rendering for JavaScript pages
Repo picker controls (TTY only):
- `Space`: check or uncheck a repo
- Mouse: hover a visible row to focus it, left-click to check or uncheck it
- `a`: check all visible repos
- `c`: clear filter and checks
- `b`: open batch-size overlay (`1`, `2`, `3`, `6`, `8`)
- `PageUp`/`PageDown`: move cursor by one viewport page
- `Home`/`End`: jump to first/last visible result
- `Enter`: clone checked repos only, or visible repos when nothing is checked
Clone dashboard controls (TTY only):
- `t`: toggle repository detail lines in the live batch display
- `q`: request stop after current batch completes
- `m`: motion effect
- `b`: Burn effect
- `w`: Wipe effect
TTY clone runs use a Scrin dashboard. Aisling renders the top Matrix deck,
effect pane, and TQDM-style loader rows for concurrent repository pulls. The TUI
is built against newer Scrin interaction support and remains keyboard-first for
SSH reliability.
Mode, provider, repository, and model picker screens also accept mouse hover and
left-click actions when the terminal supports mouse reporting.
Examples:
- Start TUI and pick target/repo interactively:
- `thesa`
- `thesa --tui`
- `thesa -T`
- Clone one repo:
- `thesa torvalds/linux --output ./repos`
- Clone all public repos for a user/org with shallow history:
- `thesa octocat --output ./repos --depth 1`
- Clone with explicit concurrency:
- `thesa octocat --concurrency 8 --output ./repos`
- Archive a website:
- `thesa archive site https://example.com --full-site --max-pages 100`
- `thesa archive url https://example.com/docs --max-depth 2`
- Verify a thesa archive directory:
- `thesa verify ./archives/example`
- `thesa verify ./archives/example --json`
- Preview actions before cloning:
- `thesa org-name --dry-run`
- Models and providers:
- `thesa --mode models --provider hf meta-llama --dry-run`
- `thesa --mode models --provider hf meta-llama/Llama-3.1-8B-Instruct --dry-run`
- `thesa --mode models --provider ollama llama3 --dry-run`
- `thesa --mode models --provider hf top --dry-run`
- `thesa --mode models --provider hf latest --dry-run`
- `thesa --mode models --provider ollama top --dry-run`
- `thesa --mode models --provider ollama latest --dry-run`
- `thesa --mode models --provider civitai top --dry-run`
- `thesa --mode models --provider civitai latest --dry-run`
- `thesa --mode models --provider civitai 827184 --dry-run`
## Scripts
You can compile and run with helper scripts:
- Run from repo root (interactive target prompt):
- `./run.sh`
- Pass a target directly:
- `./run.sh <target> [output-root] [concurrency] [extra thesa flags]`
- Compile only:
- `./scripts/compile.sh`
- Compile and run (default concurrency `4`):
- `./scripts/run.sh <target> [output-root] [concurrency] [extra thesa flags excluding --output]`
- Publish release to crates.io:
- `./scripts/publish.sh --allow-dirty`
- `./scripts/publish.sh --dry-run` (safe checks only)
`run.sh` creates an output folder from the target name (for example `owner` or,
for `owner/repo`, just `repo`) and runs clones inside that folder.
## Notes
- `--concurrency` sets the concurrent clone batch size.
- private repositories are intentionally skipped/blocked.
- Cargo packages exclude archive output, session files, logs, target directories,
and temporary test archives. Do not put secrets in project files.