thesa 4.1.18

A compendium to archive GitHub repositories, ML models, and websites
thesa-4.1.18 is not a library.

thesa

thesa is a Rust CLI to archive GitHub repositories, ML models, and websites. Website archiving is powered by siteforge.

Install

cargo install --path .

From crates.io after publishing:

cargo install thesa

Then run:

thesa

Or force the interactive TUI explicitly:

thesa --tui

Short form:

thesa -T

Usage

thesa [TARGET] [options]

New command form:

  • thesa archive site <URL>
  • thesa archive url <URL>
  • thesa archive refresh
  • thesa archive list
  • thesa archive inspect <ARCHIVE_ID>
  • thesa archive search <QUERY>
  • thesa archive export <ARCHIVE_ID> --format <FORMAT>
  • thesa archive pack <ARCHIVE_ID>
  • thesa archive verify <ARCHIVE_ID>
  • thesa archive resume <ARCHIVE_ID>
  • thesa archive diagnostics [ARCHIVE_ID]
  • thesa verify <ARCHIVE_DIR>

Run with no arguments for interactive mode.

Use --tui or -T to force interactive mode. If TARGET is also passed, the TUI flag takes precedence and opens the TUI prompt instead.

The TUI landing screen offers three archive routes: GitHub repositories, ML models, and Siteforge website archives. Selecting Siteforge prompts for a URL and then shows a preflight options screen for full-site vs single-page archives, max depth, max pages, archive output profile, and JavaScript rendering mode. TUI Siteforge runs archive into ./archives/sites by default.

TARGET can be one of: owner, owner/repo, https://github.com/owner, https://github.com/owner/repo.

For model mode, TARGET can also be presets:

  • top (most downloaded)
  • latest (most recently updated)

For model mode, set --mode models and choose one provider:

  • --provider hf (Hugging Face, default)
  • --provider ollama (local Ollama instance)
  • --provider civitai (CivitAI)

In interactive/TUI model mode, provider selection appears before the model target prompt.

Common options:

  • --depth <DEPTH>: clone with commit depth
  • -c, --concurrency <N>: number of repos cloned in parallel (default 4)
  • --filter <PATTERN>: filter repository names by substring
  • --skip-existing: skip repos that are already present
  • --token <TOKEN>: GitHub API token (or use GITHUB_TOKEN env)
  • --hf-token <TOKEN>: Hugging Face API token (or use HF_TOKEN env)
  • --civitai-token <TOKEN>: CivitAI API token (or use CIVITAI_TOKEN env)
  • --hf-mirror: use hf-mirror.com for Hugging Face API calls
  • --provider <PROVIDER>: model provider for model mode (hf, ollama, civitai)
  • -T, --tui: force the interactive Scrin interface

Git archive refresh options:

  • thesa archive refresh: scan ./archives/** for GitHub worktrees and run git pull --ff-only
  • thesa archive update and thesa archive pull: aliases for refresh
  • --archive-root <DIR>: root to scan recursively (default ./archives)
  • -c, --concurrency <N>: number of repositories refreshed at once (default 4)
  • --filter <TEXT>: refresh only matching owner/repo slugs or paths
  • --dry-run: print discovered archives without pulling

Refresh only touches actual Git worktrees. It derives owner/repo from each GitHub origin URL when available and falls back to the archive path layout, such as archives/<owner>/<repo>. Dirty worktrees are skipped with an error so local edits are not overwritten silently.

Website archive options:

  • -o, --output <DIR>: output root for website archives (default ./archives/sites)
  • --input <FILE>: seed URLs from a file, one per line (# comments allowed); may be used with or without a positional URL
  • --full-site: follow in-scope site links instead of archiving one page
  • --allow-cross-domain: permit cross-domain links
  • --max-depth <N>: limit crawl depth
  • --max-pages <N>: limit page count
  • -c, -j, --concurrency <N>: number of concurrent fetch workers
  • --delay-ms <N>: politeness delay between requests per worker
  • --archive-id <ID>: stable archive directory id
  • --include <GLOB>: include URL glob pattern, repeatable
  • --exclude <GLOB>: exclude URL glob pattern, repeatable
  • --archive-profile <full|agent|lean>: choose Siteforge artifact volume
  • --max-page-size-bytes <N>: maximum bytes fetched per HTML page
  • --max-asset-size-bytes <N>: maximum bytes fetched per asset
  • --max-total-archive-size-bytes <N>: maximum bytes written by the archive
  • --render-js: use Siteforge browser rendering for JavaScript pages
  • --render-js-auto: render only thin JavaScript app shells
  • --browser-command <COMMAND>: browser command template for JavaScript rendering
  • --browser-profile <DIR>: browser profile directory for JavaScript rendering
  • --render-wait-ms <N>: milliseconds to wait before extracting rendered DOM
  • --header 'Name: Value': extra request header, repeatable
  • --cookie <COOKIE_HEADER>: cookie header value for authenticated/public crawls
  • --user-agent <VALUE>: Siteforge fetch/render User-Agent
  • --timeout-secs <N>: HTTP timeout per request
  • --retry-count <N>: retry attempts for failed frontier items
  • --ocr: enable Siteforge OCR for code-like downloaded assets
  • --dry-run: print the Siteforge crawl plan without fetching pages

Avoid storing secrets in docs or shell history when using --header or --cookie; prefer a short-lived shell variable or a local wrapper script outside the repository.

Siteforge archives write an AI-readable bundle under the archive id directory:

  • AGENTS.md: first file to open for agent-readable instructions/context
  • agent-index.json: machine-readable index for agents/tools
  • readable/markdown, readable/text, readable/json: extracted page content
  • chunks/chunks.jsonl: chunk stream for downstream indexing
  • manifest.json, checksums.json, crawl.db: Siteforge crawl metadata
  • .thesa/manifest.json, .thesa/checksums.blake3: thesa integrity sidecars

Siteforge management commands default to ./archives/sites. Use --archive-root <DIR> when archives live elsewhere.

Siteforge export formats include markdown, basic-markdown, jsonl, json, html, warc, zip, tar, agent-tar, and all.

Repo picker controls (TTY only):

  • Space: check or uncheck a repo
  • Mouse: hover a visible row to focus it, left-click to check or uncheck it
  • a: check all visible repos
  • c: clear filter and checks
  • b: open batch-size overlay (1, 2, 3, 6, 8)
  • PageUp/PageDown: move cursor by one viewport page
  • Home/End: jump to first/last visible result
  • Enter: clone checked repos only, or visible repos when nothing is checked

Clone dashboard controls (TTY only):

  • t: toggle repository detail lines in the live batch display
  • q: request stop after current batch completes
  • m: motion effect
  • b: Burn effect
  • w: Wipe effect

TTY clone runs use a Scrin dashboard. Aisling renders the top Matrix deck, effect pane, and TQDM-style loader rows for concurrent repository pulls. The TUI is built against newer Scrin interaction support and remains keyboard-first for SSH reliability.

Mode, provider, repository, and model picker screens also accept mouse hover and left-click actions when the terminal supports mouse reporting.

Examples:

  • Start TUI and pick target/repo interactively:
    • thesa
    • thesa --tui
    • thesa -T
  • Clone one repo:
    • thesa torvalds/linux --output ./repos
  • Clone all public repos for a user/org with shallow history:
    • thesa octocat --output ./repos --depth 1
  • Clone with explicit concurrency:
    • thesa octocat --concurrency 8 --output ./repos
  • Refresh existing GitHub archives:
    • thesa archive refresh --dry-run
    • thesa archive refresh --archive-root ./archives --concurrency 8
    • thesa archive pull --filter octocat
  • Archive a website:
    • thesa archive site https://example.com --full-site --max-pages 100
    • thesa archive url https://example.com/docs --max-depth 2
    • thesa archive site https://docs.example.com --archive-profile agent --render-js-auto
    • thesa archive --dry-run url https://example.com/docs --full-site
    • thesa archive url https://example.com/docs --dry-run --include '**/docs/**' --delay-ms 500
    • thesa archive url https://example.com/docs --input seeds.txt --archive-id docs
    • thesa archive url --input seeds.txt --full-site --archive-id docs
  • Manage Siteforge archives:
    • thesa archive list
    • thesa archive inspect docs
    • thesa archive search "install" --archive-id docs --limit 5
    • thesa archive export docs --format agent-tar --output ./exports/docs-agent
    • thesa archive pack docs --max-tokens 80000
    • thesa archive verify docs
    • thesa archive resume docs
    • thesa archive diagnostics docs
  • Verify a thesa archive directory:
    • thesa verify ./archives/example
    • thesa verify ./archives/example --json
  • Preview actions before cloning:
    • thesa org-name --dry-run
  • Models and providers:
    • thesa --mode models --provider hf meta-llama --dry-run
    • thesa --mode models --provider hf meta-llama/Llama-3.1-8B-Instruct --dry-run
    • thesa --mode models --provider ollama llama3 --dry-run
    • thesa --mode models --provider hf top --dry-run
    • thesa --mode models --provider hf latest --dry-run
    • thesa --mode models --provider ollama top --dry-run
    • thesa --mode models --provider ollama latest --dry-run
    • thesa --mode models --provider civitai top --dry-run
    • thesa --mode models --provider civitai latest --dry-run
    • thesa --mode models --provider civitai 827184 --dry-run

Scripts

You can compile and run with helper scripts:

  • Run from repo root (interactive target prompt):

    • ./run.sh

  • Pass a target directly:

    • ./run.sh <target> [output-root] [concurrency] [extra thesa flags]

  • Compile only:

    • ./scripts/compile.sh

  • Compile and run (default concurrency 4):

    • ./scripts/run.sh <target> [output-root] [concurrency] [extra thesa flags excluding --output]

  • Publish release to crates.io:

    • ./scripts/publish.sh --allow-dirty
    • ./scripts/publish.sh --dry-run (safe checks only)

run.sh creates an output folder from the target name (for example owner or, for owner/repo, just repo) and runs clones inside that folder.

Notes

  • --concurrency sets the concurrent clone batch size.
  • private repositories are intentionally skipped/blocked.
  • Cargo packages exclude archive output, session files, logs, target directories, and temporary test archives. Do not put secrets in project files.