repolith

Declarative orchestrator for Rust toolchains spread across multiple sibling git repositories.

⚡ Parallel · 🛑 Cancellation-aware · 💾 Cache-first

repolith reads one repolith.toml describing remote repositories and the actions to run against them — git clone, cargo install, docker build — then executes the plan in parallel layers, with content-addressed caching so untouched work never re-runs and a shared CancellationToken so Ctrl-C cleanly aborts every in-flight subprocess.

What it looks like

Given a manifest:

[orchestrator]
schema_version = "0.1"
name = "my-stack"

[[node]]
id = "shared-types"
git = "https://github.com/example-org/shared-types"
path = "../libs/shared-types"
  [[node.action]]
  kind = "git-clone"

[[node]]
id = "migration-tool"
git = "https://github.com/example-org/migration-tool"
path = "../tools/migration-tool"
  [[node.action]]
  kind = "git-clone"
  [[node.action]]
  kind = "cargo-install"
  crate = "migrate"
  install_to = "~/.local/bin"

You preview, then sync:

$ repolith status
+-------------------------------+--------+---------------+
| Action                        | Status | Reason        |
+===============================+========+===============+
| shared-types::git-clone::0    | stale  | NoCachedBuild |
| migration-tool::git-clone::0  | stale  | NoCachedBuild |
| migration-tool::cargo-install | stale  | NoCachedBuild |
+-------------------------------+--------+---------------+

$ repolith sync --dry-run --explain
• shared-types::git-clone::0: NoCachedBuild
• migration-tool::git-clone::0: NoCachedBuild
• migration-tool::cargo-install::1: NoCachedBuild
dry-run: 3 action(s) would run

$ repolith sync
OK   shared-types::git-clone::0 (55 ms)
OK   migration-tool::git-clone::0 (188 ms)
OK   migration-tool::cargo-install::1 (4.2 s)

$ repolith sync
up to date — 0 stale actions

The second sync is a no-op — the cache holds last-run input hashes per action, so nothing re-runs unless something actually changed. What counts as "changed": the upstream HEAD, the content of a local path source (edits are seen even uncommitted), a cargo feature, the toolchain, the build platform — or the artifact going missing from this machine. That last one matters with a shared cache backend: another machine having built something is not a reason to skip building it here.

Why use it

• One declarative file. Everything your stack needs to bootstrap, in repolith.toml. No bash glue, no per-machine README dance.
• Parallel by default. tokio::FuturesUnordered + Semaphore cap concurrency at --jobs N (default = num_cpus). Layer N+1 starts only when layer N settles — typed dependencies, no race conditions.
• Cancels cleanly. A shared CancellationToken plumbed through every action's Ctx. First failure in --fail-fast (default) cancels in-flight peers; --keep-going lets the layer settle then halts. On Unix the subprocess process group is signalled (SIGTERM → grace → SIGKILL) so cargo's rustc / linker grandchildren get reaped too.
• Cache-first, and honest about it. Every successful build writes a BuildEvent keyed by a content-addressed input hash — for local path sources that means the tree's actual bytes (.gitignore-aware, target/ excluded, mtimes ignored so a fresh clone doesn't look stale). Before trusting a cache hit the planner also checks the artifact is still here. Re-runs are near-instant when nothing changed, and never skipped when something did.
• Hardened argv. URLs validated against a scheme allowlist with nested-userinfo / host / path-segment leading-dash checks; -- argv separator before every user URL as defense in depth; crate names + feature flags rejected if they could break cargo's --features list.

What it is NOT

The negative scope is fixed and will not evolve:

• Not a CI runner — no distributed execution, no remote workers.
• Not a toolchain manager — rustup is fine.
• Not a package manager — cargo is fine.
• Not a process supervisor — systemd / launchd / docker compose are fine; repolith reads heartbeats, doesn't write them.
• Not a monorepo tool — cargo workspaces already covers single-repo workspace publishing.
• Not a hermetic build system — hermeticity is opt-in per action, not the default.

Quick start

# 1. Install the binary from crates.io
cargo install repolith-cli

# 2. Write a manifest in your stack root (see the reference below,
#    or start from repolith.toml.example)
mkdir -p ~/my-stack && cd ~/my-stack
$EDITOR repolith.toml

# 3. Preview what would happen
repolith sync --dry-run --explain

# 4. Go
repolith sync

Building from source works too: git clone https://github.com/anatta-rs/repolith && cargo build --release.

repolith status prints a cache hit/miss table without running anything. repolith sync -k keeps a layer running after a failure (useful for surfacing every failure of a layer in one pass).

See repolith.toml.example for a full annotated manifest.

Manifest reference

A manifest is one [orchestrator] block plus any number of [[node]] blocks; each node carries the actions to run against it, in order.

[orchestrator]

[[node]]

Action kinds

kind = "git-clone" — fetch the node's source into path. No fields of its own:

[[node.action]]
kind = "git-clone"

kind = "cargo-install" — cargo install from the node's source tree. Every field is optional:

[[node.action]]
kind = "cargo-install"
crate = "migrate"              # default: the node's `id` — the BINARY name
package = "migration-tools"    # default: none — see below
profile = "dev"                # default: none → cargo's default, release
features = ["postgres", "tls"] # default: none
install_to = "~/.local/bin"    # default: ~/.repolith/bin (`~` expands at run time)

crate is the binary target name, which often differs from the package that contains it (this repo installs the binary repolith from the package repolith-cli).

package selects which package to build when the source holds more than one — common for git repositories shipping test fixtures or a workspace of tools, where cargo refuses to guess:

error: multiple packages with binaries found: …
Please specify a package, e.g. `cargo install --git <url> bin_only`

Leave it out unless you hit that error; omitting it lets cargo resolve on its own, which is right for single-package sources.

profile picks the cargo profile (--profile) — dev, or any profile your Cargo.toml defines. Useful for keeping debug symbols, or for a custom profile such as one with thin LTO.

Do not expect a large build-time win from dev. Measured on this repository, cold and without a compiler cache: 26.8 s release against 25.0 s dev — 7 %, for a binary 3.5× larger (11 MB → 39 MB) and 1.8× more build artifacts. Crates whose build is dominated by optimisation will differ; many are dominated by C dependencies and proc macros, which the profile does not touch. Measure before assuming.

Switching profiles re-installs: the profile is part of the input hash, since both land the binary at the same path and nothing else could tell them apart.

kind = "docker" — docker build an image from the node's checkout (build-only: running containers stays out of scope, see What it is NOT). Requires path on the node; tag is the only required field:

[[node.action]]
kind = "docker"
tag = "my-org/app:latest"      # required — docker reference charset only
dockerfile = "build/Dockerfile" # default: Dockerfile (relative to context)
context = "build"              # default: the node's `path`

dockerfile and context must stay inside the node's checkout: relative paths only, no .., validated at parse time and re-checked after symlink resolution at build time.

kind = "repolith" — federation: the node's checkout contains its own repolith.toml, executed as a nested plan (orchestrator-of-orchestrators). Requires path on the node; the one field is optional:

[[node.action]]
kind = "repolith"
manifest = "repolith.toml"     # default — relative to the node's `path`

The child stack keeps its own local cache (<stack>/.repolith/cache.db), exactly as if you had run repolith sync in that directory. Guard rails: manifest cycles (A -> B -> A) are rejected with the offending chain, federation depth is capped at 8, --jobs N bounds the whole tree (one global pool, never N per level), and Ctrl-C cancels every level down to the subprocess groups. The manifest path obeys the same two-stage containment as docker's paths.

Actions on the same node run in declaration order; independent nodes run in parallel.

Published version vs working copy

Two things you may want from the same stack: install what is released, or install what you are editing right now. repolith has no --dev flag for this — the distinction is which manifest you name, so it stays visible at the call site instead of hidden in a mode:

repolith sync                                 # ./repolith.toml — the published thing
repolith sync --manifest repolith.dev.toml    # your working copy, debug build

This repository ships both as a worked example. The pair usually differs in two places: path (a checkout repolith owns, versus your workspace) and profile (release versus dev). Switching between them rebuilds once — expected, since the profile is part of the input hash.

One trap worth stating: a node with a git-clone action resets its checkout (git fetch + git reset --hard). Never point such a node at a directory you work in; keep those under something like ~/.repolith/src and let the dev manifest be the one that reads your workspace.

Faster rebuilds with sccache

repolith runs cargo install, and cargo builds each install in a throwaway directory — so nothing is reused between syncs out of the box. A shared compiler cache fixes that, and the difference is not subtle. Measured on a small binary crate: 7.4 s cold, 1.1 s warm.

sccache is the simplest way there:

cargo install sccache --locked

# ~/.cargo/config.toml — applies to every cargo invocation, repolith included
[build]
rustc-wrapper = "sccache"

# sccache cannot cache incremental compilation; leave it off or it silently
# stops helping.
[profile.dev]
incremental = false

Cap the cache so it cannot grow without bound (SCCACHE_CACHE_SIZE="10G"), and know the limits before expecting miracles: crates with procedural macros are not cacheable, so the Rust hit rate is always partial. sccache --show-stats reports per-server-session counters that reset when the server idles out — the on-disk cache survives, only the numbers restart.

Cache backends

The build cache is pluggable (Cache trait in repolith-core). Two backends ship today; --cache (or REPOLITH_CACHE) selects one:

The Neo4j backend reads REPOLITH_NEO4J_URI, REPOLITH_NEO4J_USER, and REPOLITH_NEO4J_PASS from the environment — credentials never live in repolith.toml, and these variables are never forwarded to spawned subprocesses. Schema: one (:Action {id}) node per action with a LAST relationship to its most recent (:BuildEvent); layer writes are one transaction. NamespacedCache (library-level) lets multiple stacks share one server without id collisions.

Architecture

5 crates, layered execution with FuturesUnordered + CancellationToken + Semaphore. Full diagram + the FailFast / KeepGoing sequence diagrams

• design decisions live in ARCHITECTURE.md.

Status

M2 complete. 5 crates, 4 action kinds (git-clone, cargo-install, docker, repolith federation), 2 cache backends (SQLite default, Neo4j opt-in with a live-server contract suite in CI), parallel layered execution with tree-wide cancellation, URL-injection + path-traversal hardening. Full test suite under cargo test --workspace --all-features (110+ tests at last count). The crates.io badge above always shows the current published version; see CHANGELOG.md for the per-release breakdown. All 5 crates are published on crates.io; releases are automated with release-plz — merging the release PR publishes, tags, and cuts the GitHub Release.

Roadmap

• M2 — ✅ complete: docker action, federation kind = "repolith", Neo4j cache backend.
• M3 — watch mode (re-plan on file change), template_apply action driving AttachedEntry::Outbound.

Security

See SECURITY.md for the threat model, the env-allowlist policy, and the GitHub Security Advisories private-reporting form for vulnerability disclosure.

Contributing

PRs welcome. See CONTRIBUTING.md for the dev setup, local CI gates, and worked recipes for adding a new action or a new cache backend.

License

Dual-licensed under either of:

• Apache License, Version 2.0 (LICENSE-APACHE)
• MIT license (LICENSE-MIT)

at your option.