thesa 4.7.0

Archive repositories, models, datasets, websites, assets, audio, videos, scholarly papers, and technical reports
thesa-4.7.0 is not a library.

thesa

thesa is a Rust CLI to archive GitHub and Bitbucket repositories, ML models, datasets, websites, free 3D assets, reusable audio, exact public YouTube videos, scholarly papers, and technical reports. Repository refresh helpers live in repoforge, model target parsing lives in modelforge, dataset scraping is powered by baseforge, website archiving is powered by siteforge, license-aware 3D bundles use assetforge, and reusable music and sound effects use soundforge, scholarly documents use paperforge, and constrained YouTube video archives use videoforge.

Made by Trevor Knott for Knott Dynamics. The terminal experience is built with Scrin and Aisling for the TUI, dashboards, loaders, and visual effects.

thesa is the operator-facing binary for the Knott Dynamics archive stack. It keeps reusable parsing and repository refresh logic in focused crates, then adds CLI/TUI orchestration, archive manifests, integrity sidecars, and terminal workflows.

What It Archives

  • GitHub repositories by owner/organization and Bitbucket Cloud repositories by workspace.
  • Existing Git repository archives via safe refresh/pull workflows.
  • Hugging Face, Ollama, and CivitAI model targets.
  • Hugging Face Datasets and Zenodo dataset records through BaseForge.
  • Websites and documentation through Siteforge.
  • Poly Haven, ambientCG, Sketchfab CC0, and Google Scanned Objects 3D assets through AssetForge.
  • Wikimedia Commons music and game-oriented audio through SoundForge.
  • arXiv papers, Europe PMC open-access articles, and NASA NTRS technical reports through PaperForge.
  • Exact public YouTube video URLs through VideoForge after independent explicit authorization.

What It Produces

  • Local archive directories under ./archives by default.
  • .thesa/manifest.json and .thesa/checksums.blake3 integrity sidecars.
  • Scrin/Aisling TUI dashboards for interactive archive flows.
  • Siteforge AI-readable website bundles with AGENTS.md, indexes, compact page artifacts, and raw crawl evidence.
  • Dependency-complete, checksum-verified 3D bundles with AssetForge license manifests.
  • SHA-1-verified original audio with SoundForge attribution and license manifests.
  • Verified PDF or JATS XML documents with separate metadata and document-rights evidence.
  • One verified video media file plus videoforge-video.json, retained unchanged inside a Thesa wrapper with a separate .thesa integrity sidecar.

Install

cargo install --path .

From crates.io after publishing:

cargo install thesa

The video route additionally requires trusted local yt-dlp, ffmpeg, and ffprobe executables on PATH. Install them through trusted system packages or official distributions. Current YouTube extraction may also require a supported JavaScript runtime such as Deno and locally installed EJS components; consult the official yt-dlp installation documentation. Thesa and VideoForge do not install or update these tools and disable remote component retrieval.

On Windows, install with the Windows Rust toolchain from Git Bash, PowerShell, or Command Prompt:

rustup default stable-x86_64-pc-windows-msvc
cargo install thesa --locked

Git Bash can run the installed Windows thesa.exe, but a binary installed inside WSL is a Linux binary and will not run from Windows Git Bash. If Git Bash reports a missing DLL, check the DLL name: VCRUNTIME*.dll means the Microsoft Visual C++ Redistributable or Build Tools runtime is missing; libgcc_s_seh-1.dll or libwinpthread-1.dll usually means an MSYS2/MinGW Rust toolchain is being used, so switch to the rustup MSVC toolchain above or ensure MSYS2's mingw64/bin is on PATH. thesa uses Rustls for its direct HTTPS client and Siteforge uses bundled SQLite, so default installs do not require external OpenSSL or SQLite DLLs.

Then run:

thesa

Or force the interactive TUI explicitly:

thesa --tui

Short form:

thesa -T

Usage

thesa [TARGET] [options]

New command form:

  • thesa archive site <URL>
  • thesa archive url <URL>
  • thesa archive dataset <TARGET>
  • thesa archive asset <TARGET>
  • thesa archive audio <TARGET> (sound, sounds, and soundforge aliases)
  • thesa archive paper <QUERY> (papers and paperforge aliases)
  • thesa archive video <URL> (videos and videoforge aliases)
  • thesa archive refresh [OWNER]
  • thesa archive list
  • thesa archive inspect <ARCHIVE_ID>
  • thesa archive search <QUERY>
  • thesa archive export <ARCHIVE_ID> --format <FORMAT>
  • thesa archive pack <ARCHIVE_ID>
  • thesa archive verify <ARCHIVE_ID>
  • thesa archive resume <ARCHIVE_ID>
  • thesa archive diagnostics [ARCHIVE_ID]
  • thesa verify <ARCHIVE_DIR>

Run with no arguments for interactive mode.

Version 4.7.0 adds VideoForge inspection and explicitly authorized archival for one exact public YouTube video URL. Existing mode indexes 0..=6 are unchanged; videos are appended at index 7. Video completion sidecars are written only after the selected/archive/skip/failure evidence agrees and every returned archive has passed VideoForge verification.

Use --tui or -T to force interactive mode. If TARGET is also passed, the TUI flag takes precedence and opens the TUI prompt instead.

The TUI landing screen offers eight archive routes: GitHub/Bitbucket repositories, ML models, BaseForge datasets, Siteforge website archives, AssetForge free 3D assets, SoundForge reusable audio, PaperForge scholarly papers/reports, and VideoForge public YouTube videos. Selecting BaseForge prompts for a dataset target and archives into ./archives/datasets by default. Selecting Siteforge prompts for a URL and then shows a preflight options screen for full-site vs single-page archives, max depth, max pages, archive output profile, and JavaScript rendering mode. TUI Siteforge runs archive into ./archives/sites by default. Selecting AssetForge offers Poly Haven, ambientCG, Sketchfab CC0, or Google Scanned Objects, then prompts for an id, search, category, top, or latest target. Results default to the top 20, are ranked before selection, and archive into ./archives/assets. Selecting SoundForge first offers Music, Game Audio, or Any Audio and then prompts for a Commons file, search, category, top, or latest target. Audio archives default to ./archives/audio; Commons access is anonymous and never opens a token screen. Selecting PaperForge uses the p mnemonic and requires an arXiv, Europe PMC, or NASA NTRS context choice before query entry. It has no token screen. Collection results stay in the shared fullscreen ranked picker and an explicit empty selection is a no-op. Selecting VideoForge uses the v mnemonic and proceeds directly to an exact URL prompt without a provider context menu or discovery list. Before an actual download, fullscreen mode is restored and the user must have supplied --video-authorized or type the exact phrase I AM AUTHORIZED.

Agent Guides

thesa --agent and thesa --agent all print the Thesa guide plus the exact embedded guides shipped by every Forge crate. The route runs before TUI/target prompts, network requests, and filesystem operations and writes stable plain Markdown to stdout only. It does not print token values or environment contents.

Selectors and aliases:

  • thesa
  • repoforge, repos, repositories
  • modelforge, models
  • baseforge, datasets
  • siteforge, sites
  • assetforge, assets, 3d
  • soundforge, audio, sounds
  • paperforge, papers, paper
  • videoforge, videos, video

Examples: thesa --agent, thesa --agent videoforge, thesa --agent paperforge, and thesa --agent=sites. An unknown selector fails with the complete valid-selector list. thesa --agent --help remains normal Clap help.

Siteforge also provides siteforge --agent. The other Forge crates are library-only and expose their exact text through public AGENT_GUIDE constants; Thesa embeds those constants and does not spawn sibling executables.

Videos

VideoForge accepts one exact HTTPS YouTube watch, youtu.be, shorts, live, or embed URL containing an 11-character video ID and canonicalizes it to the matching https://www.youtube.com/watch?v=... URL. The initial release is YouTube-only: there is no discovery list, playlist mode, Twitter support, provider selection, filter, limit, arbitrary yt-dlp arguments, authentication, cookies, credentials, proxies, or DRM/geo/access-control bypass.

Command mode exposes --quality best|max-1080p|max-720p, --max-download-bytes (default 2 GiB, nonzero), --timeout-secs (default 1800, nonzero), --archive-root (default ./archives/videos), --skip-existing, --dry-run, and --authorized. Legacy mode uses --mode videos (aliases video, videoforge) and the prefixed --video-quality, --video-max-download-bytes, --video-timeout-secs, and --video-authorized flags with the shared output, skip, and dry-run options.

Examples:

  • thesa archive video 'https://www.youtube.com/watch?v=dQw4w9WgXcQ' --dry-run
  • thesa archive videos 'https://youtu.be/dQw4w9WgXcQ' --quality max-1080p --max-download-bytes 2147483648 --authorized
  • thesa --mode videos --video-quality max-720p --video-authorized 'https://www.youtube.com/shorts/dQw4w9WgXcQ'

Every dry run performs VideoForge metadata inspection but no download. It prints the canonical URL, title, uploader/channel, reported license, attribution, redistribution warning, estimated size, quality, byte cap, timeout, output, and skip behavior. Actual archival always reinspects and requires explicit authorization. In an interactive terminal, absence of the authorization flag requires the exact phrase I AM AUTHORIZED; any other input cancels without a download. Noninteractive runs must pass the flag.

Authorization is never inferred from metadata or a license label. VideoForge, Thesa, and the yt-dlp software license grant no download, redistribution, or other rights in video content. Follow the YouTube Terms of Service, applicable law, and content-owner rights. A reported Creative Commons label still requires review; a Standard YouTube License or missing license does not grant redistribution permission.

Thesa gives VideoForge an isolated provider root inside <video-id>/videoforge/. VideoForge writes one nested <video-id>/ directory containing one verified MP4, WebM, or Matroska file and videoforge-video.json. Thesa validates the complete single-target summary, calls videoforge::verify_archive for every archived or verified-skipped record, prints record warnings, then writes <video-id>/.thesa/manifest.json and checksums.blake3 only for those verified returned directories. The sidecar remains outside the exact-file VideoForge directory, so provider verification and --skip-existing remain valid. A reported failure is returned after any successful archives have been retained.

Papers And Technical Reports

Legacy mode uses --mode papers (aliases paper, paperforge), --paper-provider arxiv|europe-pmc|nasa-ntrs, optional --paper-sort relevance|date|citations, --paper-limit <1-100> (default 20), and --paper-max-download-bytes (default 256 MiB). Its default output is ./archives/papers. Command mode adds --archive-root, --skip-existing, --filter, --max-download-bytes, --limit, and --dry-run.

PaperForge queries support optional words, required +word, excluded -word, quoted phrases, and typed title:, abstract:, author:, topic:/category:, and journal: clauses. Exact identities are arxiv:, pmid:, pmcid:, ntrs:, doi:, or id: and supported canonical provider/DOI URLs. Date bounds use after:YYYY[-MM[-DD]] and before:YYYY[-MM[-DD]]. Directives are @recent, @top, @sort:relevance|date|citations, @limit:1..100, and @provider:arxiv|europe-pmc|nasa-ntrs. In command mode, --provider must match a query @provider. When --provider is omitted, the directive selects the provider; if both are omitted, arXiv is used.

Examples:

  • thesa --mode papers --paper-provider arxiv 'title:"graph neural networks" +author:kipf after:2016 @recent' --dry-run
  • thesa archive paper 'arxiv:1706.03762' --provider arxiv --dry-run
  • thesa archive papers 'topic:cancer after:2023 @sort:citations @limit:10' --provider europe-pmc
  • thesa archive paper 'topic:aeronautics after:2020' --provider nasa-ntrs --limit 20 --dry-run

Every provider fetch is bounded to at most 100 candidates before local filtering and deterministic ranking. Dry runs show examined/eligible counts, ranking tier and score, and a warning when the candidate cap may have truncated matches. A CLI sort overrides query @sort only when explicitly supplied. arXiv and NASA do not provide citation counts, so citations display as unavailable, never zero; PaperForge rejects citation sorting and @top for those providers.

Metadata rights and document rights are separate. OpenAlex/Crossref-style open metadata does not license a PDF. arXiv archives provider PDFs with a mandatory three-second API pacing gate and never infers redistribution permission. NASA archives only NTRS PDFs that pass PaperForge's document URL/access gates and likewise makes no redistribution inference. Europe PMC archives JATS XML; license and permissions extracted from the downloaded JATS can differ from search metadata, so Thesa reports final rights from ArchivedPaper.manifest after the archive. Retain source, attribution, access status, terms, and redistribution warnings for every provider.

Repository TARGET can be an owner/workspace, owner/repo, GitHub URL, or Bitbucket URL. Bare targets use --repo-provider github|bitbucket; provider URLs and SSH remotes are detected automatically.

Examples:

  • thesa torvalds/linux --repo-provider github --dry-run
  • thesa workspace/repo --repo-provider bitbucket --dry-run
  • thesa https://bitbucket.org/workspace/repo --dry-run

For model mode, TARGET can also be presets:

  • top (most downloaded)
  • latest (most recently updated)

For model mode, set --mode models and choose one provider:

  • --provider hf (Hugging Face, default)
  • --provider ollama (local Ollama instance)
  • --provider civitai (CivitAI)

In interactive/TUI model mode, provider selection appears before the model target prompt.

For dataset mode, TARGET can be:

  • owner for a Hugging Face dataset namespace.
  • owner/dataset or dataset:<id> for one Hugging Face dataset.
  • search:<query> for a Hugging Face dataset search.
  • a Hugging Face dataset URL such as https://huggingface.co/datasets/squad.
  • a Zenodo record id or https://zenodo.org/records/<id> when using --dataset-provider zenodo.
  • top or latest presets.

For dataset mode, set --mode datasets and choose one provider:

  • --dataset-provider hf (Hugging Face Datasets, default)
  • --dataset-provider zenodo (Zenodo records)

The command form is also available:

  • thesa archive dataset dataset:squad --provider hf --dry-run
  • thesa archive dataset "climate data" --provider zenodo --dry-run

For free 3D asset mode, provider targets include:

  • a provider id or canonical provider URL
  • search:<query> or category:<name>
  • top or latest

Bundle controls:

  • --asset-format auto|gltf|fbx|blend|usd|archive and --asset-resolution auto|original|1k|2k|4k|8k in top-level mode
  • --format and --resolution under thesa archive asset
  • --asset-limit <1-100> or --limit <1-100> controls ranked discovery (default 20)
  • --asset-max-download-bytes or --max-download-bytes to bound each selected bundle
  • --skip-existing verifies every local size and BLAKE3 digest, plus provider MD5 when supplied

Poly Haven assets are CC0, but access to api.polyhaven.com is governed by separate terms and is free only for reasonable non-commercial use unless Poly Haven grants a commercial API license. Thesa displays required Poly Haven attribution and the API terms URL next to API-derived content. ambientCG assets are CC0 and use its official API. Sketchfab searches are limited to downloadable CC0 models; downloads require --asset-token or SKETCHFAB_API_TOKEN and preserve creator plus Sketchfab attribution. Google Scanned Objects uses the official Gazebo Fuel API without a token. The collection is CC BY 4.0 and requires retained Google Research attribution, which AssetForge records with the license and source metadata. Thesa prints the exact attribution text Google Research via Gazebo Fuel for both archive runs and dry runs.

For audio mode, targets include bare searches, search:<query>, category:<Commons category>, file:<Commons file title>, Commons file URLs, top, and latest. Use --audio-kind music|game-audio|any, --audio-limit <1-50>, and --audio-max-download-bytes in top-level mode. The command form uses --kind, --limit, --max-download-bytes, --archive-root, --skip-existing, --filter, and --dry-run.

SoundForge uses Wikimedia Commons' official Action API and original upload.wikimedia.org files; it does not scrape Commons pages. Its commercial-capable allowlist is CC0, verified public domain, CC BY, and CC BY-SA. Thesa retains creator attribution, the source page, normalized license, and license URL even where attribution is not legally required. Copyright or public-domain status does not remove trademark, privacy, publicity, personality, cultural, or other rights, so review each source page for the intended use.

Asset and audio dry runs list selected archive candidates. With --skip-existing, provider manifest and checksum verification happens during the actual archive operation, so a candidate shown by a dry run may then be safely skipped. AssetForge and SoundForge collection discovery and ranked result selection remain in the fullscreen TUI. Filters are applied before the picker; exact and single-result targets proceed directly. Command and non-TUI runs retain the numbered selector.

Common options:

  • --depth <DEPTH>: clone with commit depth
  • -c, --concurrency <N>: number of repos cloned in parallel (default 4)
  • --filter <PATTERN>: filter repository names by substring
  • --skip-existing: skip repos that are already present
  • --token <TOKEN>: GitHub API token (or use GITHUB_TOKEN env)
  • --bitbucket-token <TOKEN>: Bitbucket access token (or use BITBUCKET_TOKEN env)
  • --repo-provider <PROVIDER>: provider for bare repository targets (github, bitbucket)
  • --hf-token <TOKEN>: Hugging Face API token (or use HF_TOKEN env)
  • --civitai-token <TOKEN>: CivitAI API token (or use CIVITAI_TOKEN env)
  • --hf-mirror: use hf-mirror.com for Hugging Face API calls
  • --provider <PROVIDER>: model provider for model mode (hf, ollama, civitai)
  • --dataset-provider <PROVIDER>: dataset provider for dataset mode (hf, zenodo)
  • --asset-provider polyhaven|ambientcg|sketchfab|google-scanned-objects: free 3D asset provider
  • --asset-token <TOKEN>: user Sketchfab API token (or SKETCHFAB_API_TOKEN)
  • --asset-format <FORMAT> / --asset-resolution <RESOLUTION>: AssetForge bundle selection
  • -T, --tui: force the interactive Scrin interface

Git archive refresh options:

  • thesa archive refresh: scan ./archives/** for GitHub/Bitbucket worktrees and run git pull --ff-only
  • thesa archive refresh <NAMESPACE>: refresh only archives whose slug owner/workspace matches <NAMESPACE>
  • thesa archive update and thesa archive pull: aliases for refresh
  • --archive-root <DIR>: root to scan recursively (default ./archives)
  • --owner <NAMESPACE> / --workspace <NAMESPACE>: provider namespace filter
  • -c, --concurrency <N>: number of repositories refreshed at once (default 4)
  • --filter <TEXT>: refresh only matching owner/repo slugs or paths
  • --dry-run: print discovered archives without pulling

Refresh only touches actual Git worktrees. It derives owner/repo from recognized GitHub or Bitbucket origin URLs and falls back to the archive path layout, such as archives/<owner>/<repo>. Dirty worktrees are skipped with an error so local edits are not overwritten silently. The discovery, slugging, clean-check, and git pull --ff-only execution are provided by the repoforge crate; thesa adds archive manifests and terminal reporting around that core.

Namespace-scoped refresh is the fastest way to update one provider archive:

  • thesa archive refresh tknott95 --dry-run
  • thesa archive refresh tknott95 --concurrency 8
  • thesa archive pull --owner tknott95 --filter agent

Archive output directories are ignored by git and excluded from Cargo packages.

Website archive options:

  • -o, --output <DIR>: output root for website archives (default ./archives/sites)
  • --input <FILE>: seed URLs from a file, one per line (# comments allowed); may be used with or without a positional URL
  • --full-site: follow in-scope site links instead of archiving one page
  • --allow-cross-domain: permit cross-domain links
  • --max-depth <N>: limit crawl depth
  • --max-pages <N>: limit page count
  • -c, -j, --concurrency <N>: number of concurrent fetch workers
  • --delay-ms <N>: politeness delay between requests per worker
  • --archive-id <ID>: stable archive directory id
  • --include <GLOB>: include URL glob pattern, repeatable
  • --exclude <GLOB>: exclude URL glob pattern, repeatable
  • --archive-profile <full|agent|lean>: choose Siteforge artifact volume
  • --max-page-size-bytes <N>: maximum bytes fetched per HTML page
  • --max-asset-size-bytes <N>: maximum bytes fetched per asset
  • --max-total-archive-size-bytes <N>: maximum bytes written by the archive
  • --render-js: use Siteforge browser rendering for JavaScript pages
  • --render-js-auto: render only thin JavaScript app shells
  • --browser-command <COMMAND>: browser command template for JavaScript rendering
  • --browser-profile <DIR>: browser profile directory for JavaScript rendering
  • --render-wait-ms <N>: milliseconds to wait before extracting rendered DOM
  • --header 'Name: Value': extra request header, repeatable
  • --cookie <COOKIE_HEADER>: cookie header value for authenticated/public crawls
  • --user-agent <VALUE>: Siteforge fetch/render User-Agent
  • --timeout-secs <N>: HTTP timeout per request
  • --retry-count <N>: retry attempts for failed frontier items
  • --ocr: enable Siteforge OCR for code-like downloaded assets
  • --dry-run: print the Siteforge crawl plan without fetching pages

Avoid storing secrets in docs or shell history when using --header or --cookie; prefer a short-lived shell variable or a local wrapper script outside the repository.

Siteforge archives write an AI-readable bundle under the archive id directory. The default agent profile avoids duplicate YAML Markdown and plain-text copies:

  • AGENTS.md: first file to open for agent-readable instructions/context
  • agent-index.json: machine-readable index for agents/tools
  • readable/basic_markdown, readable/json: default extracted page content
  • readable/markdown, readable/text: additional representations in explicit full mode
  • chunks/chunks.jsonl: chunk stream for downstream indexing
  • manifest.json, checksums.json, crawl.db: Siteforge crawl metadata
  • .thesa/manifest.json, .thesa/checksums.blake3: thesa integrity sidecars

Siteforge management commands default to ./archives/sites. Use --archive-root <DIR> when archives live elsewhere.

Siteforge export formats include markdown, basic-markdown, jsonl, json, html, warc, zip, tar, agent-tar, and all.

Forge Crates

thesa keeps reusable archive logic in small crates so it can stay focused on CLI/TUI orchestration:

  • repoforge: discovers Git worktrees under archive roots, derives GitHub owner/repo slugs, rejects dirty worktrees, and refreshes clean clones with git pull --ff-only.
  • modelforge: normalizes Hugging Face, Ollama, and CivitAI target strings into provider-neutral target enums before thesa calls provider APIs.
  • baseforge: discovers Hugging Face/Zenodo datasets, inspects downloadable files, and archives them with bounded concurrent downloads.
  • siteforge: captures websites into AI-readable local archives and provides website archive management commands.
  • assetforge: discovers ranked license-aware models through official Poly Haven, ambientCG, and Sketchfab APIs plus CC BY 4.0 Google Scanned Objects through Gazebo Fuel, and writes bounded, BLAKE3-verified archives.
  • soundforge: discovers allowlisted reusable audio through the official Wikimedia Commons Action API and writes bounded SHA-1/BLAKE3-verified archives.
  • paperforge: performs bounded rights-aware discovery and verified archival of arXiv PDFs, Europe PMC JATS XML, and NASA NTRS technical-report PDFs.
  • videoforge: constrains yt-dlp to exact public YouTube URLs and writes bounded, explicitly authorized, verified single-video archives.

The thesa binary combines those crates with Scrin and Aisling to provide the interactive command deck, repo/model/dataset pickers, clone dashboards, Siteforge option screens, AssetForge discovery, and archive progress displays.

Package Safety

The Cargo package is explicitly whitelisted. Releases include only the manifest, lockfile, README, license, changelog, source, and helper scripts. Archive output, sessions, notes, logs, build directories, and local scratch files are not shipped.

Included package files:

  • Cargo.toml
  • Cargo.lock
  • README.md
  • LICENSE
  • CHANGELOG.md
  • AGENT_GUIDE.md
  • src/**
  • git.sh
  • scripts/**

Repo picker controls (TTY only):

  • Space: check or uncheck a repo
  • Mouse: hover a visible row to focus it, left-click to check or uncheck it
  • a: check all visible repos
  • c: clear filter and checks
  • b: open batch-size overlay (1, 2, 3, 6, 8)
  • PageUp/PageDown: move cursor by one viewport page
  • Home/End: jump to first/last visible result
  • Enter: clone checked repos only, or visible repos when nothing is checked

Asset/audio/paper ranked picker controls (fullscreen TUI):

  • Up/Down: move through ranked results
  • Space: toggle the active result
  • a: select all ranked results
  • n: clear all results
  • Enter: confirm the current selection; an empty selection is a no-op
  • Esc: cancel the route

Clone dashboard controls (TTY only):

  • t: toggle repository detail lines in the live batch display
  • q: request stop after current batch completes
  • m: motion effect
  • b: Burn effect
  • w: Wipe effect

TTY clone runs use a Scrin dashboard. Aisling renders the top Matrix deck, effect pane, and TQDM-style loader rows for concurrent repository pulls. The TUI is built against newer Scrin interaction support and remains keyboard-first for SSH reliability.

Mode, provider, repository, and model picker screens also accept mouse hover and left-click actions when the terminal supports mouse reporting. Dataset selection uses a compact numbered prompt after BaseForge discovery.

Examples:

  • Start TUI and pick target/repo interactively:
    • thesa
    • thesa --tui
    • thesa -T
  • Clone one repo:
    • thesa torvalds/linux --output ./repos
  • Clone all public repos for a user/org with shallow history:
    • thesa octocat --output ./repos --depth 1
  • Clone with explicit concurrency:
    • thesa octocat --concurrency 8 --output ./repos
  • Refresh existing GitHub archives:
    • thesa archive refresh --dry-run
    • thesa archive refresh tknott95 --dry-run
    • thesa archive refresh --archive-root ./archives --concurrency 8
    • thesa archive pull --filter octocat
    • thesa archive pull --owner tknott95 --concurrency 8
  • Archive a website:
    • thesa archive site https://example.com --full-site --max-pages 100
    • thesa archive url https://example.com/docs --max-depth 2
    • thesa archive site https://docs.example.com --archive-profile agent --render-js-auto
    • thesa archive --dry-run url https://example.com/docs --full-site
    • thesa archive url https://example.com/docs --dry-run --include '**/docs/**' --delay-ms 500
    • thesa archive url https://example.com/docs --input seeds.txt --archive-id docs
    • thesa archive url --input seeds.txt --full-site --archive-id docs
  • Manage Siteforge archives:
    • thesa archive list
    • thesa archive inspect docs
    • thesa archive search "install" --archive-id docs --limit 5
    • thesa archive export docs --format agent-tar --output ./exports/docs-agent
    • thesa archive pack docs --max-tokens 80000
    • thesa archive verify docs
    • thesa archive resume docs
    • thesa archive diagnostics docs
  • Verify a thesa archive directory:
    • thesa verify ./archives/example
    • thesa verify ./archives/example --json
  • Preview actions before cloning:
    • thesa org-name --dry-run
  • Models and providers:
    • thesa --mode models --provider hf meta-llama --dry-run
    • thesa --mode models --provider hf meta-llama/Llama-3.1-8B-Instruct --dry-run
    • thesa --mode models --provider ollama llama3 --dry-run
    • thesa --mode models --provider hf top --dry-run
    • thesa --mode models --provider hf latest --dry-run
    • thesa --mode models --provider ollama top --dry-run
    • thesa --mode models --provider ollama latest --dry-run
    • thesa --mode models --provider civitai top --dry-run
    • thesa --mode models --provider civitai latest --dry-run
    • thesa --mode models --provider civitai 827184 --dry-run
  • Datasets and providers:
    • thesa --mode datasets --dataset-provider hf dataset:squad --dry-run
    • thesa --mode datasets --dataset-provider hf search:"code data" --filter python --dry-run
    • thesa --mode datasets --dataset-provider hf openai --concurrency 8 --skip-existing
    • thesa archive dataset dataset:squad --provider hf --archive-root ./archives/datasets
    • thesa archive dataset "climate data" --provider zenodo --dry-run
  • Free 3D assets:
    • thesa --mode assets --asset-provider polyhaven --asset-resolution 1k Shelf_01 --dry-run
    • thesa --mode assets --asset-format gltf --asset-resolution 2k search:"wooden chair"
    • thesa archive asset category:furniture --format fbx --resolution 1k --dry-run
    • thesa archive asset Shelf_01 --format gltf --resolution 1k --skip-existing
    • thesa archive asset search:chair --provider gso --dry-run
  • Reusable audio:
    • thesa --mode audio --audio-kind music top --dry-run
    • thesa archive audio search:laser --kind game-audio --limit 10 --dry-run
    • thesa archive sounds category:"Sound effects" --kind any --skip-existing
  • Exact public YouTube videos:
    • thesa archive video 'https://www.youtube.com/watch?v=dQw4w9WgXcQ' --dry-run
    • thesa archive videoforge 'https://youtu.be/dQw4w9WgXcQ' --quality max-1080p --authorized
    • thesa --mode videos --video-authorized 'https://www.youtube.com/watch?v=dQw4w9WgXcQ'

Scripts

You can compile and run with helper scripts:

  • Run from repo root (interactive target prompt):

    • ./run.sh

  • Pass a target directly:

    • ./run.sh <target> [output-root] [concurrency] [extra thesa flags]

  • Compile only:

    • ./scripts/compile.sh

  • Compile and run (default concurrency 4):

    • ./scripts/run.sh <target> [output-root] [concurrency] [extra thesa flags excluding --output]

  • Publish release to crates.io:

    • ./scripts/publish.sh --allow-dirty
    • ./scripts/publish.sh --dry-run (safe checks only)

run.sh creates an output folder from the target name (for example owner or, for owner/repo, just repo) and runs clones inside that folder.

Notes

  • --concurrency sets the concurrent clone batch size.
  • private repositories are intentionally skipped/blocked.
  • Cargo packages exclude archive output, session files, logs, target directories, and temporary test archives. Do not put secrets in project files.