thesa 4.1.6

A compendium to archive GitHub repositories, ML models, and websites
thesa-4.1.6 is not a library.

thesa

thesa is a Rust CLI to archive GitHub repositories, ML models, and websites. Website archiving is powered by siteforge.

Install

cargo install --path .

From crates.io after publishing:

cargo install thesa

Then run:

thesa

Or force the interactive TUI explicitly:

thesa --tui

Short form:

thesa -T

Usage

thesa [TARGET] [options]

New command form:

  • thesa archive site <URL>
  • thesa archive url <URL>
  • thesa verify <ARCHIVE_DIR>

Run with no arguments for interactive mode.

Use --tui or -T to force interactive mode. If TARGET is also passed, the TUI flag takes precedence and opens the TUI prompt instead.

The TUI landing screen offers three archive routes: GitHub repositories, ML models, and Siteforge website archives. Selecting Siteforge prompts for a URL and runs a same-domain full-site crawl into ./archives/sites by default.

TARGET can be one of: owner, owner/repo, https://github.com/owner, https://github.com/owner/repo.

For model mode, TARGET can also be presets:

  • top (most downloaded)
  • latest (most recently updated)

For model mode, set --mode models and choose one provider:

  • --provider hf (Hugging Face, default)
  • --provider ollama (local Ollama instance)
  • --provider civitai (CivitAI)

In interactive/TUI model mode, provider selection appears before the model target prompt.

Common options:

  • --depth <DEPTH>: clone with commit depth
  • -c, --concurrency <N>: number of repos cloned in parallel (default 4)
  • --filter <PATTERN>: filter repository names by substring
  • --skip-existing: skip repos that are already present
  • --token <TOKEN>: GitHub API token (or use GITHUB_TOKEN env)
  • --hf-token <TOKEN>: Hugging Face API token (or use HF_TOKEN env)
  • --civitai-token <TOKEN>: CivitAI API token (or use CIVITAI_TOKEN env)
  • --hf-mirror: use hf-mirror.com for Hugging Face API calls
  • --provider <PROVIDER>: model provider for model mode (hf, ollama, civitai)
  • -T, --tui: force the interactive Scrin interface

Website archive options:

  • --full-site: follow in-scope site links instead of archiving one page
  • --allow-cross-domain: permit cross-domain links
  • --max-depth <N>: limit crawl depth
  • --max-pages <N>: limit page count
  • --include <GLOB>: include URL glob pattern, repeatable
  • --exclude <GLOB>: exclude URL glob pattern, repeatable
  • --render-js: use Siteforge browser rendering for JavaScript pages

Repo picker controls (TTY only):

  • Space: check or uncheck a repo
  • Mouse: hover a visible row to focus it, left-click to check or uncheck it
  • a: check all visible repos
  • c: clear filter and checks
  • b: open batch-size overlay (1, 2, 3, 6, 8)
  • PageUp/PageDown: move cursor by one viewport page
  • Home/End: jump to first/last visible result
  • Enter: clone checked repos only, or visible repos when nothing is checked

Clone dashboard controls (TTY only):

  • t: toggle repository detail lines in the live batch display
  • q: request stop after current batch completes
  • m: motion effect
  • b: Burn effect
  • w: Wipe effect

TTY clone runs use a Scrin dashboard. Aisling renders the top Matrix deck, effect pane, and TQDM-style loader rows for concurrent repository pulls. The TUI is built against newer Scrin interaction support and remains keyboard-first for SSH reliability.

Mode, provider, repository, and model picker screens also accept mouse hover and left-click actions when the terminal supports mouse reporting.

Examples:

  • Start TUI and pick target/repo interactively:
    • thesa
    • thesa --tui
    • thesa -T
  • Clone one repo:
    • thesa torvalds/linux --output ./repos
  • Clone all public repos for a user/org with shallow history:
    • thesa octocat --output ./repos --depth 1
  • Clone with explicit concurrency:
    • thesa octocat --concurrency 8 --output ./repos
  • Archive a website:
    • thesa archive site https://example.com --full-site --max-pages 100
    • thesa archive url https://example.com/docs --max-depth 2
  • Verify a thesa archive directory:
    • thesa verify ./archives/example
    • thesa verify ./archives/example --json
  • Preview actions before cloning:
    • thesa org-name --dry-run
  • Models and providers:
    • thesa --mode models --provider hf meta-llama --dry-run
    • thesa --mode models --provider hf meta-llama/Llama-3.1-8B-Instruct --dry-run
    • thesa --mode models --provider ollama llama3 --dry-run
    • thesa --mode models --provider hf top --dry-run
    • thesa --mode models --provider hf latest --dry-run
    • thesa --mode models --provider ollama top --dry-run
    • thesa --mode models --provider ollama latest --dry-run
    • thesa --mode models --provider civitai top --dry-run
    • thesa --mode models --provider civitai latest --dry-run
    • thesa --mode models --provider civitai 827184 --dry-run

Scripts

You can compile and run with helper scripts:

  • Run from repo root (interactive target prompt):

    • ./run.sh

  • Pass a target directly:

    • ./run.sh <target> [output-root] [concurrency] [extra thesa flags]

  • Compile only:

    • ./scripts/compile.sh

  • Compile and run (default concurrency 4):

    • ./scripts/run.sh <target> [output-root] [concurrency] [extra thesa flags excluding --output]

  • Publish release to crates.io:

    • ./scripts/publish.sh --allow-dirty
    • ./scripts/publish.sh --dry-run (safe checks only)

run.sh creates an output folder from the target name (for example owner or, for owner/repo, just repo) and runs clones inside that folder.

Notes

  • --concurrency sets the concurrent clone batch size.
  • private repositories are intentionally skipped/blocked.
  • Cargo packages exclude archive output, session files, logs, target directories, and temporary test archives. Do not put secrets in project files.