rch — Remote Compilation Helper
|
Transparent remote compilation for multi-agent development
TL;DR
Problem: Many concurrent AI agents can saturate local CPU and make your workstation unusable.
Solution: RCH runs as a Claude Code PreToolUse hook, classifies build-like commands in milliseconds, executes them on remote workers, and returns artifacts/output as if they ran locally.
Design constraint: RCH is fail-open. If remote execution is not safe/possible, commands run locally.
What RCH Intercepts
RCH currently recognizes and can offload:
| Ecosystem | Intercepted Commands |
|---|---|
| Rust | cargo build, cargo check, cargo clippy, cargo doc, cargo test, cargo nextest run, cargo bench, rustc |
| Bun/TypeScript | bun test, bun typecheck |
| Go | go build -o <file>, ordinary go test, go vet |
| C/C++ | gcc, g++, clang, clang++ |
| Build Systems | make, cmake --build, ninja, meson compile |
| Nix | nix build, nix-build, nix flake check, nix develop -c <cmd>, nix shell -c <cmd> |
Nix builds only route to workers that advertise a nix capability (a usable nix
binary plus a populated /nix/store); on a fleet with no such worker they fall
back to local execution (or are refused under RCH_REQUIRE_REMOTE=1, exactly as
with Bun/Node). Nix outputs stay in the worker's /nix/store behind a result
symlink, so these run as streaming, exit-status-only commands (no artifacts are
copied back — the flake source is synced out, the build runs, the result stays
remote).
Go builds support an explicit, project-relative file output, for example
go build -o bin/app ./cmd/app or go build -o 'products/app [dev]*?' main.go.
The output is returned to that exact path, including when CARGO_TARGET_DIR is
set. Common build flags such as -p, -tags, -trimpath, and -race are
supported, as are linker stripping flags (-ldflags '-s -w') and -X variable
definitions. Builds require the durable Unix execution path and a local Go
installation. Both endpoints must report empty effective GOFLAGS and the
dispatcher's native GOOS/GOARCH. The caller's selected Go version, CGO
configuration, experiments, and architecture tuning settings are captured
before upload and must match the worker exactly before compilation starts.
Ordinary builds with CGO enabled remain supported when those settings match.
The worker compiles into a fresh private file outside the source root, preserving
wildcard go:embed inputs. Only after successful compilation produces a regular
file does it create output parents and an adjacent stage for atomic replacement
of the previous worker file. Retrieval then stages and validates that file before
replacing the local output. A missing download cannot be satisfied by an old
local binary, and durable recovery resumes collection without rerunning Go.
Directory outputs, symlinks or symlinked output parents, cross compilation,
native-library modes such as c-shared/c-archive, and low-level flags that can
create extra files are outside this output contract. Implicit forms such as
go build, go build ., and go build ./... stay local because their output
names depend on which packages they select. Go test options that write binaries,
profiles, or fuzz corpora (-c, -o, -coverprofile, -cpuprofile, -trace,
-fuzz, and related flags, including -test.* forms after -args) also stay
local. Strict remote mode refuses
unsupported forms instead of executing them locally.
RCH explicitly does not intercept local-mutating or interactive patterns (examples):
- Package management:
cargo install,cargo clean,bun install,bun add,bun remove - Bun runners/dev:
bun run,bun build,bun dev,bun x/bunx - Nix interactive/mutating: bare
nix develop/nix shell,nix run,nix repl,nix profile,nix flake update,nix store gc,nix-env,nix-shell - Watch/background/piped/redirected commands where deterministic offload is unsafe
Why It Works Well
- Transparent hook behavior: agents see normal command semantics.
- 5-tier classification pipeline optimized for very fast non-compilation rejection.
- Daemon-owned worker state and slot accounting.
- Cache-aware worker selection and project affinity.
- Queue + cancellation primitives for overloaded scenarios.
- Deterministic reliability subsystems for convergence, pressure handling, and remediation.
- Unified status surface with posture and remediation hints.
Current Architecture
Agent Shell / Claude Code
|
v
PreToolUse Hook -> rch (classifier + hook protocol)
|
v
rchd (daemon)
- worker selection
- queueing and cancellation metadata
- health, alerts, telemetry, history
- reliability subsystems (convergence, pressure, triage)
|
v
Remote workers (rch-wkr)
- execute build/test commands
- manage worker cache
- report capabilities/health/telemetry
Workspace crates:
rch/: Hook + primary CLIrchd/: Local daemon + scheduling/reliability APIsrch-wkr/: Worker execution/caching agentrch-common/: Shared protocol/types/patterns/UI foundationsrch-telemetry/: Telemetry collection/storage integration
Reliability Model (Operational)
RCH now includes a deterministic reliability stack for multi-repo and multi-worker stability:
- Path-dependency closure planning: builds can include required repository closure rather than a single root.
- Canonical topology enforcement: worker/project roots are normalized around
/data/projectsand/dpconventions. - Repo convergence service: tracks worker drift vs required repos and can repair drift.
- Disk pressure resilience: pressure scoring, admission control, safe reclaim with active-build protection.
- Process triage/remediation: bounded TERM/KILL escalation with audit trail.
- Cancellation orchestration: deterministic cancellation metadata and worker health integration.
- Unified posture/reporting: status output includes posture, convergence state, pressure, and actionable remediation hints.
Installation
Recommended: Installer
|
From Source
Source Build Note
All dependencies — including the FrankenTUI (ftui-*), rich_rust, and TOON
(tru) crates — resolve from crates.io, so a clean git clone && cargo build
builds on any machine with no special directory layout or pre-cloned
dependency tree required.
rsync Requirement (macOS note)
RCH moves source and artifacts with rsync over SSH and works best with
rsync 3.2 or newer (--info=progress2 progress, zstd compression,
--append-verify resumes). On Linux the distro rsync qualifies.
On macOS, the stock /usr/bin/rsync is openrsync (macOS 15+) or Apple's
rsync 2.6.9 (older releases); neither accepts rsync 3.x flags. RCH handles this
automatically: it probes rsync --version, prefers a Homebrew/MacPorts rsync
(/opt/homebrew/bin/rsync, /usr/local/bin/rsync, /opt/local/bin/rsync)
even when it is not first on PATH, and otherwise drives the stock binary with
a compatible flag set (--progress --stats -vv, zlib instead of zstd, and the
zero-build-output detector fails open). For full-speed transfers:
rch doctor reports which binary and flavour RCH resolved. To pin one
explicitly, set [transfer] rsync_bin = "/path/to/rsync" or export
RCH_RSYNC_BIN.
First-Time Setup
Fastest Path
rch init can guide:
- Worker discovery from SSH config/aliases
- Worker probing and selection
rch-wkrdeployment- Toolchain synchronization
- Daemon startup
- Hook installation
- Validation build
Manual Path
# 1) configure workers
# 2) start daemon
# 3) verify workers
# 4) install hook
# 5) check posture
Command Surface
Global flags:
|
||
Core Operations
|||||
|||||||
||||
|
||||
||
On macOS, daemon autostarts defer to a registered com.rch.daemon launchd
service without restarting its running process. The managed daemon waits if
another daemon still owns its socket. A private daemon bypasses this service
discovery when given both a nondefault socket and an explicit workers
configuration; a custom socket alone cannot be delegated to the shared service.
Fleet GC reports progress on stderr as workers finish. Each worker has a
15-minute budget shared across connections, scanned roots, and collection
batches; set rch gc --worker-timeout=120 for a shorter budget. A timeout makes
the command exit nonzero while preserving completed workers' results. JSON/TOON
output contains one final response with those results and the error.
On timeout, GC terminates its foreground SSH process and allows two additional
seconds to reap it; cleanup failures are reported. This does not confirm whether
a remote collection batch finished: an interrupted --apply batch is reported
as having an unknown outcome, alongside confirmed earlier batches. The default
invocation remains a preview.
Capability refreshes skip duplicate requests for a worker while its probe is
running. Workers reuse verified Rustup inventory across requests and invalidate
entries when installed toolchains or component metadata change. Inventory runs
in bounded batches; unfinished or failed entries appear in probe_warnings,
and their component capabilities remain unavailable until verified. Disk, load,
and project-topology observations are collected again for each request.
Hook + Agent Integration
|||
|||||
|
Named Cargo JSON for installer consumers
An installer selecting explicit binary targets can request caller-visible Cargo JSON through the canonical Cargo shim:
RCH_CARGO_JSON_STDOUT=1 RCH_REQUIRE_REMOTE=1
This opt-in requires the durable POSIX worker route and refuses local fallback,
arbitrary jobs, machine envelopes, and unsupported target selections. Selected
binary records reach stdout after artifact publication and acknowledged worker
release. Their executable, filenames, manifest, and source paths refer to the
caller tree. Selected output files are copied into an invocation-private retained
directory beneath the caller's actual target root and checked against the exact
publication fingerprints. A later build cannot replace these copies by publishing
into the shared target paths. Each record carries an rch extension retaining the original worker
record, exact retained stdout receipt, wrapper/build identity, and executable
BLAKE3 publication fingerprint. Unselected dependency records keep their worker
metadata and do not assert local delivery of intermediates. Diagnostics stay on
stderr. Compiler failure stdout comes from the same invocation's immutable
nonzero completion receipt; a successful compile with failed artifact delivery
advertises no caller-local success records. The ordinary hook and rch --json exec envelope contracts are unchanged.
The delivery JSON, its private file paths, and their fingerprints are persisted before source retirement and worker release. After the original wrapper has exited, recover that same invocation without compiling again:
This explicit output mode requires an invocation originally admitted with
RCH_CARGO_JSON_STDOUT=1; it cannot upgrade an ordinary job. It returns verified
retained stdout only after the original recovery and acknowledgement complete,
and uses the original exit status. Repeating it returns the first delivery's
exact JSON bytes and private paths, including after later builds or a failed
stdout write. A live unacknowledged wrapper or unknown owner identity refuses
detached recovery. --json machine envelopes conflict with --cargo-json.
The complete compiler stderr is collected from the same supervisor log with
the original completion status and retained before retirement. Its path,
length, BLAKE3 and compiler status appear in the selected record's receipt.
Recovery verifies those bytes and writes them to stderr without line or UTF-8
conversion. Failed compiler stdout and stderr use the exact nonzero completion;
no successful Cargo record is advertised when artifact delivery failed. Older
completed journals lacking retained diagnostics refuse this explicit output
mode rather than fabricate diagnostic parity.
Use the exit status together with the record binding. This route does not
provide native Windows delivery, an installer transaction, a source-content
receipt, or performance qualification. scripts/e2e_cargo_json_stdout.py checks
the actual shim, retained producer bytes, both real native binaries, cached
delivery, completed same-id stdout recovery, a broken stdout pipe, and a genuine
Cargo target-selection failure against an externally
admitted source and artifact root. No test result is implied by this description.
Wrapper-loss, pending recovery, cancellation, full installers, native Windows,
and same-invocation live-incumbent/A/A performance acceptance remain open.
Job Mode (non-compilation workloads)
rch exec --job admits an arbitrary NON-compilation workload (sharded tests,
fuzzing, benchmarks, mutation testing) onto the same remote rails. It bypasses
ONLY the compilation classifier — the PreToolUse hook never sets it, so
auto-delegation of ordinary commands is impossible.
# Declared result directories sync back on ANY exit code (including failures):
Semantics: the remote exit status surfaces verbatim; no toolchain/worker-env
rerun heuristics apply. A declared --result-dir that is missing or only
partially transferable fails loudly (RCH-E309, exit 102) regardless of the
job's own exit status. Paths must be repository-relative; conflicts with
--clean-overlay / --source-content-receipt are refused.
Native Apple commands can use job mode too: direct xcodebuild and xcrun
invocations require a worker declaring os = "darwin". This preserves the OS
fence: ordinary jobs cannot consume that worker, and Linux workers cannot take
an Xcode job. Invoke the tool directly; RCH does not inspect arbitrary scripts
or infer a Mac requirement from text merely mentioning Xcode.
RCH_REQUIRE_REMOTE=1 RCH_BUILD_SLOTS=1
This is host routing, not proof that Xcode, signing credentials, or a Simulator
are ready. Native test and Simulator safety checks still apply. Both rch and
rchd need the native-command routing support; keep explicit strict-remote
mode when qualifying a build so an older daemon cannot silently run it locally.
Requiring a verified tool
A compilation or job that needs a tool the fleet does not uniformly have can demand a worker that has verified it:
The gate runs before any slot is reserved, and it is evidence-based: the worker must have run the operator-declared probe (see Worker Config Example) successfully. A project can make the requirement a default for all of its jobs:
[]
= ["clang"]
Project defaults ADD to --require-tool in job mode. These [jobs] defaults
apply only to jobs; explicit CLI requirements also apply to compilations while
preserving their classifier, Rust target and toolchain checks. Any required tool
makes execution strict remote, including when configuration forces local or
disables RCH. Both client and daemon must support the verified-tools endpoint;
an older daemon refuses the request instead of silently ignoring requirements.
A successful named probe establishes cached tool presence. It does not bind the executable bytes, compiler version or SDK inventory used by a particular build.
rch admit [--job] [--require-tool NAME]... -- <command> preflights the same
decision read-only — it syncs nothing and reserves nothing. Without --job the
preflight consults the compilation classifier, so a fuzz script answers
local; with it the classifier is bypassed exactly as exec --job does, and
the answer is offload. The classification facts are still reported, so a
compilation accidentally passed to --job still shows its family.
This static preflight reports requirements and classification; it does not
verify current worker readiness or establish native release qualification.
A required name that no worker has verified admits no worker at all,
including when the name is a typo. That is deliberate: silently dropping the
requirement would route the job to a machine that cannot run it, and job mode
returns the remote exit status verbatim — so the resulting failure would be
indistinguishable from the job's own. Selection reports
capability_missing:tool:<name>:probe_failed when a worker declared the tool
but its probe failed, and ...:not_declared when nothing declared it.
Same-identity job recovery
rch --json jobs reports valid leases alongside journal_errors. An unreadable
or malformed journal, or an entry observed as nonregular, makes complete false
and the command exits 1, while valid jobs remain visible in the same JSON document.
Each valid row
also reports owner_presence as live, absent or unknown; a false legacy
wrapper_alive value alone does not establish that an owner is absent. The listing
does not remove, repair or treat the unreadable entry as an absent owner. An
initially missing journal directory is an empty, complete listing.
The directory is opened once and journal names are read relative to its file
descriptor. Journal opens reject symlinks and do not wait for a FIFO writer;
the opened descriptor must represent a regular file before any bytes are read.
The directory root itself must be a directory rather than a symlink.
Initial attach/recover loads and recovery-owner reloads use the same regular-file,
nofollow, nonblocking reader. A changed parent directory is refused; a FIFO
cannot block these journal reads while waiting for a writer.
Detached recovery retains the opened directory and persistent lock through
journal publication and terminal reporting. Its clones share that ownership;
reads and atomic writes use the held directory. A replaced directory or lock
refuses acknowledgement. Failed recovery publications retain their exclusive
temporary bytes for diagnosis; recovery does not remove them. Ordinary running
wrappers keep their existing journal writer.
complete describes this scan, not an immutable ownership snapshot: entries
and regular-file contents can still change during observation.
If a journal path contains non-UTF-8 bytes, its error has a printable path
and an exact Unix path_bytes array. A job command whose output cannot be written
or flushed exits with an error rather than reporting successful delivery.
Completed/recovered reports include an admission reference from one committed
terminal observation: exact decimal build ID, wrapper, worker, journal path and
caller fingerprint, matching the ordinary execution reference. Unadmitted
terminal jobs have admission: null. Unknown exit, incomplete identity or
unacknowledged state cannot become a completed report. This reports journal
identity/custody; it does not replay the original execution envelope or attest
compiler, SDK, slots or artifact contents.
These commands never replay the original command. Recovery requires retained
identity and completion/retrieval evidence; ambiguous or missing evidence is
reported as an error. For a live wrapper stalled during artifact retrieval,
recover_requested means the request was recorded, not that outputs have arrived:
use jobs attach to observe completion. The wrapper cancels and reaps the old
transfer before retrying that retrieval phase. An absent wrapper can recover
outstanding outputs from its retained journal without starting another build.
Use a rebuilt daemon with the identity-aware job routes; older daemons are refused.
On POSIX workers, source ownership is saved before synchronization starts and remains active through output retrieval, including after the SSH lock holder disappears. Another job cannot overwrite an overlapping source root while that ownership is unresolved; disjoint source trees can still run concurrently. Uploads, verification, execution, and collection check the same ownership token. Operations within one token are serialized. An uncertain preparation failure ends that preparation; its token is drained and cancelled before a new attempt, so an older upload cannot arrive after verification and change compiler inputs. Recovery of an interrupted preparation drains surviving transfers and cancels that token, so a delayed upload cannot mutate a tree after ownership is released. Once execution may have started, recovery requires its exact completion record. If a worker loses its ownership registry or the local recovery recipe predates this protocol, recovery refuses to infer ownership from the current source bytes. Worker cache cleanup also respects retained source grants, including jobs whose outputs are quiet but still await recovery.
Config + Diagnostics
||||||||||
|
|||
Fleet + Release + UX
|||||
|
|
|||
Agent Discovery Surface
For AI agents and automation, start with:
--json uses the standard API envelope for command output; --format toon
emits the same data as TOON. The legacy rch --capabilities flag remains a
raw JSON shortcut for lightweight discovery.
Configuration
Primary files:
- User config:
~/.config/rch/config.toml - Worker list:
~/.config/rch/workers.toml - Project override:
.rch/config.toml - Optional project excludes:
.rchignore
Precedence (highest first):
- CLI flags
- Environment variables
- Profile defaults
.env/.rch.env- Project config
- User config
- Built-in defaults
Canonical Project Root
By default rch expects projects to live under /data/projects (with the /dp
alias symlink). Both roots are configurable, so repos under ~/code,
/workspace, etc. work without relocating anything:
[]
= "/home/me/code" # default: /data/projects
= "/home/me/code" # default: /dp (set equal to canonical_root if you have no alias)
Or via the CLI / environment (env wins over TOML):
# or
Empty strings are treated as unset (defaults apply). See
docs/guides/configuration.md ([path_topology]) for details.
Minimal Example
[]
= true
= false
= false
= "~/.cache/rch/rch.sock"
= "info"
[]
= 0.85
= 2000
= 1.2
= 4
= 8
= 2
# Optional additional build-disk headroom for each remote job (GiB).
# Include expected output growth and a safety margin. Zero disables it.
# disk_headroom_gib = 80
= 300
= 1800
= 600
= true
[]
= 3
# Staging base for the transfer paths that do NOT mirror the client's layout:
# `--clean-overlay` roots, Windows workers, and `rch cache warm`. An ordinary
# `rch exec` mirrors the project under the worker's `[path_topology]
# canonical_root` instead, with the pooled Cargo target store inside that
# mirror — see `[remediation.pooled_target] store_base` below to place those
# stores on another filesystem.
= "/data/tmp/rch"
# Optional per-attempt source-sync cap. When unset, the default is payload-aware:
# 30 seconds plus one second per MiB, capped at one hour.
# sync_timeout_ms = 120000
# Abort a source sync after this many seconds of NO rsync output (dead channel /
# wedged rsync); a progressing transfer is never affected. 0 disables.
# source_sync_silence_timeout_secs = 120
# Explicit rsync binary. Unset lets RCH prefer a modern (3.x) rsync over the
# stock macOS openrsync automatically; RCH_RSYNC_BIN overrides this.
# rsync_bin = "/opt/homebrew/bin/rsync"
= true
= false
= 2048
[]
# Where pooled Cargo target stores are PLACED on the worker. Unset (the
# default) keeps them inside the project mirror, i.e. under the canonical
# root, on whatever filesystem holds it. Set an absolute path to move every
# pooled store to <store_base>/<project_id>/.rch-target-<worker>-pool-<key>,
# which is how a worker with a small root disk keeps multi-GB warm pools on a
# larger volume. `rch gc`, `rch cache status`, and the daemon sweep scan this
# root in addition to `remote_base` (with `reaper_max_cache_gb` applying per
# scan root). Ignored for Windows workers.
# store_base = "/bigdisk/rch-pools"
#
# Where the reaper SCANS for stale target dirs (not a placement setting).
# remote_base = "/data/projects"
#
# EXTRA roots `rch gc` scans, on top of the ones it derives itself
# (`remote_base`, `store_base`, and the worker temp base resolved exactly as
# the code that creates the dirs resolves it: $TMPDIR -> /data/tmp -> /tmp).
# Use this for a runtime root rch cannot know about, e.g. an operator build
# root on a mounted volume. Each entry must be absolute, `..`-free and free of
# shell metacharacters -- it is embedded in a remote command. `rch gc --root`
# adds one for a single run.
# gc_extra_roots = ["/mnt/big/rch"]
#
# Idle days before `rch gc` may collect a durable per-worker Cargo cache
# (`rch-cargo-cache-*`). 0 disables. Collection ALSO requires zero open file
# descriptors under the dir and no live process rooted at it; a gate that
# cannot be evaluated counts as "in use".
# gc_cargo_cache_idle_days = 14
[]
= "balanced"
# Reserve this much free disk, then budget space for each concurrent slot (GiB).
= 10.0
= 10.0
[]
= true
= true
[]
= true
= 300
When the Unix launcher enforces a compilation deadline, RCH preserves exit 137
and reports that the configured deadline expired. It does not retry on a larger
worker or fall back locally. Machine-mode rch exec reports
outcome: "deadline_exceeded". This classification requires evidence from the
launcher; exit 137 or elapsed time alone does not establish a timeout.
For projects with large build outputs, set compilation.disk_headroom_gib in
.rch/config.toml. A value of 80 requires 80 GiB of additional space on the
worker's reported build filesystem. Selection requires a successful disk probe
within 90 seconds and subtracts budgets already held by active builds in this
daemon. Budgets survive daemon restart and remain held until the owning build
completes. Releasing a budget does not prove its output files freed any space:
the next budgeted admission requires a disk probe started after that completion.
A probe already in flight cannot reuse the earlier free-space reading. Retries,
queue recovery, and rch diagnose use the same requirement.
Smaller CPU-slot estimates and cache affinity cannot bypass it. Older daemons
reject the distinct budgeted selection endpoint rather than ignoring the
requirement.
This is admission accounting for the worker's reported canonical/alias build
roots, not a filesystem quota or a measurement of arbitrary custom target
mounts. Other dispatchers, undeclared jobs, and external writes are outside its
accounting. The full budget stays reserved even after a disk sample reflects
some of the build's output, so admission deliberately errs toward leaving extra
space. The default is 0, retaining ordinary disk-pressure admission.
Without a declaration, the daemon still learns each project's footprint. For
every remote build that had a worker to itself (no other build from this
daemon overlapped there), it records how far the worker's free build-disk
space fell between the admission probe and the lowest probe seen while the
build ran. Footprints are kept per project and command class (cargo test is
learned separately from cargo check), for 30 days, in
history.footprints.json beside the build history. When some candidate
worker has room for the largest recent footprint plus 10% (at least 5 GiB),
after declared budgets and the remaining growth of builds already running
there, selection only considers those workers. If none has room, selection is
unchanged. Learned footprints are evidence, not a budget, so they never refuse
a build. Other dispatchers' builds can inflate a measurement and cache cleanup
can shrink one; declare disk_headroom_gib when you need a hard requirement.
Unix artifact downloads estimate the files matched by the retrieval filters
before transferring them. Their default total retry budget grows with that size
(30 seconds plus one second per MiB, adjusted for a slower bwlimit_kbps, up to
one hour). An explicit max_transfer_time_ms keeps its existing hard ceiling
and skips estimation. Rsync also stops after 30 seconds without network I/O.
If estimation fails, RCH logs the failure and retains the configured retry
budget. A literal cargo build --bin NAME or --example NAME retrieves only
the named payloads, not the pooled target's other cached outputs. Anything the
narrowing cannot parse exactly (unknown flags, shell expansion, other target
kinds, response files, custom target specs) keeps the broad per-profile filters,
which can include other cached build outputs in the same profile.
Windows workers return artifacts through tar in both quiet and interactive
execution. The transfer honors the requested output patterns and cache exclusions,
including custom Cargo target directories and explicit compiler outputs outside
target/. RCH inventories regular files, downloads only selected paths, and
validates the complete archive before writing outputs locally. Remote tar errors,
missing selected files, and unexpected archive members fail artifact retrieval;
successful downloads include file counts and a manifest for output validation.
Inventory and transfer use bounded deadlines, and cancellation reaps the transport
before recovery can begin. This transport requires GNU find from the worker's
Git for Windows installation and native tar.
Artifact staging retains the original caller directory as the source-protection reference. Recovery restores that reference for each output phase, so an empty staging directory cannot make source files eligible for a broad output pattern.
Built-in worker selection defaults to balanced, which blends speed, load,
health, and cache affinity. Use priority only when you want explicit
worker-priority control, and fair_fastest when you want extra load spreading.
Worker capacity is capped at the configured total_slots and reduced to
floor((free_disk_gb - min_free_gb) / disk_gb_per_slot) as disk fills, with a
minimum of zero. Unknown disk telemetry keeps the configured ceiling. Existing
builds retain their reservations when capacity drops; new work waits for space.
rch workers list reports the effective total when the daemon is reachable.
Change the budget with rch config set selection.disk_gb_per_slot 10 and restart
the daemon to apply it.
Balanced selection also favors free disk before a worker reaches the admission
floor. selection.weights.disk (default 0.2, range 0 to 1) adds disk
headroom credit after the other score adjustments. Equal ample headroom preserves
the existing worker order. Credit rises from zero at min_free_gb to full credit
at 25% free disk, with at least one slot's disk budget above the floor on small
disks. Unknown measurements receive full credit; the admission gate still handles
missing telemetry. Affinity pins and explicit worker requests retain their
existing behavior. Set the weight to zero to disable disk ranking independently
of disk admission and slot limits, then restart the daemon to apply the change.
rch workers list shows daemon-reported free disk and pressure. rch workers probe
shows fresh disk measurements alongside latency and labels any cached daemon
pressure separately. JSON and TOON include numeric free GiB and free ratios;
unavailable measurements remain null (displayed as unknown in text).
Worker Config Example
[[]]
= "css"
= "203.0.113.20"
= "ubuntu"
= "~/.ssh/id_rsa"
= 32
= 100
= ["fast", "ssd"]
# Verified named tools. Each entry is a FIXED argv the worker runs as the
# configured user; a zero exit marks the tool present, anything else marks it
# absent. Unlike `tags`, which are an unverified naming convention, these are
# evidence — which is what lets `--require-tool` gate worker selection.
= [
{ = "clang", = ["clang", "--version"] },
{ = "ld.lld", = ["/usr/bin/ld.lld", "--version"] },
]
Names must be ASCII letters, digits, -, _, . or +: a name carrying a
space or = could not survive the probe's fact format and would be read back
as a different name, so it is refused when the config loads rather than
becoming a capability lie. rch workers capabilities --refresh reports the
results under Named tools as verified: and failed: — a declared probe
that fails is a different operational state from one that was never declared,
and only the first points at a broken worker.
There is deliberately no general remote-shell probe API. A caller that could ask a worker to run an arbitrary command "to check for a tool" would be a remote execution primitive wearing a capability-probe hat; declaring the argv in operator config keeps the set of probe commands finite and reviewable.
Output Modes
RCH auto-selects output mode by context:
hook: strict JSON for hook protocolmachine: explicit machine output (--json,--format)interactive: rich terminal renderingcolored: ANSI-only when forced without TTYplain: text fallback
Environment controls:
RCH_JSON=1,RCH_HOOK_MODE=1NO_COLOR=1,FORCE_COLOR=1,FORCE_COLOR=0RCH_OUTPUT_FORMAT=json|toon,TOON_DEFAULT_FORMAT
JSON responses use a stable envelope (api_version, timestamp, success, data, error).
Placement Controls
Worker placement, strict-remote, queue, wait-timeout, visibility, and target-dir
behavior are first-class, canonical controls — not folklore. The authoritative
list is discoverable at runtime (rch capabilities --json), and the resolved
plan for any command is shown by rch diagnose <command> --json under
data.placement (and in the human Placement Controls section):
| Control | Env (aliases) | Effect |
|---|---|---|
| Requested worker | RCH_WORKER (RCH_WORKERS) |
Request specific worker(s) by id. Still passes capability/admission checks; an inadmissible requested worker is refused with a stable RCH-Innn reason code and a next action — never silently swapped. |
| Requested profile | RCH_PRESET |
Named execution profile (recorded as requested_profile). |
| Strict remote (fail-closed) | RCH_REQUIRE_REMOTE |
Refuse local fallback (proof mode). Takes precedence over RCH_FORCE_REMOTE. |
| Force remote (fail-open) | RCH_FORCE_REMOTE |
Always attempt offload (bypass local-time/speedup gating) but still fail open to local. Distinct from RCH_REQUIRE_REMOTE. |
| Queue when busy | RCH_QUEUE_WHEN_BUSY (default 1) |
Wait for a busy worker instead of falling back to local. Set 0 to disable. |
| Wait timeout | RCH_DAEMON_WAIT_RESPONSE_TIMEOUT_SECS (RCH_DAEMON_RESPONSE_TIMEOUT_SECS) |
Max seconds to wait for a queued worker. |
| Visibility | RCH_VISIBILITY=none|summary|verbose (RCH_QUIET, RCH_VERBOSE) |
Hook output verbosity. |
| Target dir | RCH_DISABLE_TARGET_REUSE |
Legacy unique-per-job remote target dir instead of the pooled, reuse-friendly dir. |
| Source-sync timeout | RCH_SYNC_TIMEOUT_MS |
Per-attempt source-upload timeout in milliseconds (1000..=3600000). Unset uses the payload-aware default. This does not change remote Cargo or artifact-return timeouts. |
The resolved plan reports requested_worker, requested_profile,
effective_worker, strict_remote_policy, queue_policy, visibility_mode,
wait_timeout_ms, target_dir_policy, the requested-worker admissibility
outcome, and a diagnostics list. Any control value that cannot be applied as
written (an unrecognized value or a superseded alias) surfaces a diagnostic
rather than being silently ignored.
Monitoring and Observability
RCH exposes observability through daemon APIs and metrics:
- daemon health/readiness endpoints
- Prometheus metrics collection
- opt-in OTLP export of daemon request-duration metrics
- telemetry-backed worker SpeedScore history
- queue/build history, active alerts, cancellation metadata in status APIs
- an opt-in tailnet status API (
[api] bind = "tailscale"): the daemon's full/statusJSON over TCP, bearer-token gated, bound only to loopback or Tailscale addresses — so agents on other machines can ask a dispatcher "why are you local-only right now?" without ssh - the fleet dashboard under
dashboard/(encrypted static console + an agent endpointGET /api/fleet?view=problems|diagnose|help);rch webopens it
Enable OTLP metrics in the daemon's environment with RCH_OTEL_ENABLED=1
and RCH_OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 (OTLP/gRPC).
The endpoint falls back to OTEL_EXPORTER_OTLP_ENDPOINT. OTEL_SERVICE_NAME
defaults to rchd; RCH_OTEL_EXPORT_INTERVAL_SECS defaults to 30.
rch_request_duration_seconds{entrypoint="rchd_api"} records finite Unix-socket
request handling after parsing, including errors and cancellation. Event streams
and the separate HTTP API are excluded. The same observations reach Prometheus,
including when OTLP is disabled. Shutdown drains request tasks before a
best-effort final export. The banner reports exporter configuration, not collector
connectivity.
The same opt-in settings enable direct OTLP verdict export for single-shot
rch doctor --reliability, including under --quiet. Its service name defaults
to rch. Each report emits one rch_doctor_verdict_total observation, labeled
with the actual verdict and scope (combined scopes use other), and flushes
before returning its original exit code. Reports whose daemon_unreachable
flag is true also increment rch_doctor_daemon_unreachable_total once, even
when several diagnostics report the missing daemon. The asynchronous
daemon_status and repo_convergence probes also emit
rch_doctor_probe_duration_seconds{probe,result} once on completion, RPC error,
or timeout. Durations measure each probe's execution, not time spent waiting
to join other probes. Skipped probes emit nothing; panic/cancellation and
helper/ownership probe durations are not yet exported. With --fix, each
remediation step also emits rch_doctor_fix_steps_total and
rch_doctor_fix_duration_seconds, labeled by its actual outcome: applied,
already_satisfied, would_apply, manual, or failed. Dry-run previews
remain would_apply; they are never counted as applied changes. Durations
measure step execution, excluding the initial shared configuration load.
These CLI metrics are not forwarded to the daemon's Prometheus endpoint.
Hook/watch telemetry, remaining doctor event metrics, and trace/log export
remain unfinished.
Quick checks:
On Linux boxes configured with [general] role = "dispatcher", status and doctor
also report unmanaged local Cargo/rustc processes. The warning remains visible
while those processes run, including when the daemon is unavailable. Check
rch shim status, PATH order, and absolute-path toolchain Cargo invocations.
Processes carrying RCH_CARGO_WRAPPER_BYPASS=1 or descended from rch are excluded.
An alarm event is logged once per observed episode and clears after a successful
scan finds no unmanaged compiler processes. Status and doctor share a locked
one-byte latch at $XDG_CACHE_HOME/rch/local-build-alarm (normally
~/.cache/rch/local-build-alarm). Scan failures preserve the previous episode;
latch failures are reported without hiding current builds. Reliability doctor
includes this check in the triage scope and its default all scope.
Testing and Validation
Workspace checks:
Reliability and E2E suites are provided under:
tests/tests/e2e/rch-common/tests/(contract/reliability/perf suites)
If you are running CPU-intensive validation manually and want explicit offload:
When local fallback is not acceptable, set RCH_REQUIRE_REMOTE=1 on the
rch exec process and keep the build command as direct argv:
RCH_REQUIRE_REMOTE=1
RCH_REQUIRE_REMOTE=1
Exact source-content receipts
Proof-oriented callers can require a single-worker, fail-closed source transfer and retain the exact regular-file bytes admitted on that worker:
RCH_REQUIRE_REMOTE=1
Receipt mode resolves the active Cargo path-dependency closure, stages its
roots beneath the configured worker build base, transfers with checksum
comparison, and reopens every selected file before and after the command.
Preflight initializes the isolated staging topology and skips canonical worker
repository convergence. The emitted
rch.source_content_receipt.v1 JSON binds the worker and build IDs, exact
command digest and exit code, transfer filter policy, per-root file paths,
lengths, executable bits, SHA-256 digests, and content roots. Any unproved
dependency closure, transfer delta, worker verification error, local lasting
change, retry, or local fallback refuses the invocation instead of emitting a
receipt. Receipt mode currently requires the Unix rsync transport and cannot be
combined with clean-overlay mode.
On Linux with target reuse enabled, supported Cargo commands use an owned
source/target pair. The source layout preserves sibling dependencies, selected
manifests, and enclosing workspace metadata under a stable
source-content-pair-<key>/projects/ container. Every invocation materializes
fresh source files there while holding the pair and its complete source grant.
Uploads use checksums without deleting entries or running cache sweeps; the
read-only rsync barrier still rejects surplus selected files. Cargo's target
and build directories are bound to the granted pool outside that container.
An unchanged file can reuse its previous compiler-visible timestamp only when
the previous attempt completed its source-content verification. An interrupted
or unverified attempt forces fresh timestamps beyond the pool's artifacts on
the next invocation. After artifact delivery, the source container moves to
its recorded .retained-<attempt> sibling using Linux no-replace rename. The
job journal retains both paths before acquisition, so recovery can reconcile
an interrupted move without replacing an existing retained tree. Python 3 and
the Linux rename capability are checked before source ownership is acquired.
This owned route validates Cargo manifests and configuration against the
captured transfer inventory. Relative and absolute --manifest-path selectors
map into that inventory. File-based --config, custom target-specification
files, directory-changing wrappers, and caller overrides of managed output
paths or Cargo home currently refuse before execution. Other Unix platforms,
invocations with RCH_DISABLE_TARGET_REUSE=1, and external Cargo drivers such
as Nextest, Zigbuild, and xwin retain invocation-unique staging paths. The
owned route currently covers the built-in build, test, check, documentation,
benchmark, and verified packaging families, plus Clippy.
The receipt is emitted after the remote command and its post-command source barrier, but before ordinary artifact retrieval completes. Artifact-grade callers must therefore retain both the receipt and the later successful retrieval/terminal transcript; the receipt alone is not proof that build artifacts reached the caller. A recursive local mutation watcher is still needed when transient edit-and-restore (ABA) detection is part of the caller's source-stability contract.
Clean Git overlays for shared working trees
rch exec can build an immutable committed tree plus an explicit, repeatable
set of local paths without transferring unrelated working-tree changes:
RCH_REQUIRE_REMOTE=1
RCH_REQUIRE_REMOTE=1
The client resolves --base to a commit object, streams that Git archive into
a fresh isolated worker path, and then uploads only the literal repository-
relative --overlay-path selections. Modified and untracked files are
supported; selected deletions, absolute/traversing paths, Git metadata,
non-ASCII/backslash/control-character paths, case-only or otherwise ambiguous
filesystem spellings, overlay symlinks, empty overlay directories, submodules,
and Git archive export-ignore/export-subst attributes fail closed. Exactly
one of one-or-more --overlay-path options or --no-overlay is required.
Clean-overlay mode also implies remote-only execution even when
RCH_REQUIRE_REMOTE was omitted, so a worker or transfer failure never falls
back to the ambient local tree.
Explicit clean-overlay execution also admits the read-only cargo fmt --check
diagnostic, which the ordinary interception classifier intentionally leaves
local.
On Unix workers, pooled Cargo builds use a stable source path paired with a
versioned target cache. Jobs sharing that pair wait until the preceding job
has finished execution, artifact retrieval, and source retirement. Different
pairs remain independent. Source timestamps are refreshed beyond cached
artifacts so changed archive or overlay contents cannot appear older than a
previous build; registry and Git dependency caches remain reusable.
Clean-overlay execution also pins Cargo's build.build-dir to its assigned
target directory, overriding ambient and command-line Cargo configuration.
Explicit --build-dir options are refused. This placement applies to isolated
builds too, so disabling target reuse cannot inherit shared intermediates.
Managed placement currently supports built-in Cargo build commands and Clippy.
Other compiling Cargo subcommands, including Nextest and Zigbuild, are refused
in clean-overlay mode.
An interrupted pair retains its ownership marker and refuses further reuse,
even if its SSH lock connection disappears. Inspect the prior job before
recovering that pair, or use RCH_DISABLE_TARGET_REUSE=1 for a fresh isolated
build. Non-Cargo Windows clean-overlay jobs also use isolated targets. The ownership
marker protects against process interruption; it is not a power-loss durability
guarantee.
Clean-overlay materializes the primary Git repository and explicitly bound
sibling repositories. In-repository Cargo workspace members are present in the
archive. Before Cargo starts, selected
manifests, configuration, symlinks, and command-line path overrides are checked
against the selected Git base and overlays. Escaping paths are refused with
RCH-E413; retained sibling directories on the worker cannot supply those
dependencies. To include a sibling Git repository, bind its revision explicitly:
Repeat --dependency-base PATH=REV for every external root, including transitive
dependencies. Each revision resolves to a commit before worker selection; dirty
files in those repositories are excluded. The selected repositories must be
siblings of the primary Git root. They are staged under one owned container,
preserving ../dep references, and the execution receipt names each commit,
Git tree hash, and remote path. All roots share the same execution lease and
are retired together. Cross-repository symlinks and non-sibling layouts are
currently refused.
This check conservatively covers all selected manifests, including inactive
fixtures. File-based --config, configuration includes, and changes to Cargo's
working directory are refused. On the worker, any Cargo-home or ancestor
configuration is refused before Cargo starts, even if that configuration would
be harmless. Cargo clean-overlay jobs currently require a Unix worker for this
check. These checks do not sandbox arbitrary file reads by build scripts.
The client re-fingerprints overlays after upload and refuses execution if their
contents changed during admission or transfer.
Do not batch several Cargo commands behind a shell wrapper such as
rch exec -- bash -lc "cargo test ... && cargo test ...". Shell-wrapped
commands are classified as non-compilation for hook safety. Under
RCH_REQUIRE_REMOTE=1, RCH refuses that local fallback before executing the
shell and reports RCH-E301; without the env var, ordinary non-compilation
commands may still run locally. For several focused checks, run separate direct
RCH_REQUIRE_REMOTE=1 rch exec -- cargo ... invocations.
Security Model
- Transport uses SSH.
- Worker commands are constrained to classified execution paths.
- Sensitive field masking and structured error taxonomy are built in.
rch updateenforces the release's SHA-256 checksums and verifies the archive's.minisigagainst a public key pinned in the binary (minisign key69B3955C8D2E62A8, which signs every release since 2.0.0); a bad signature aborts the update. A Sigstore bundle is also verified when the release ships one andcosignis installed.--skip-verifybypasses signature checks only.install.shstill verifies checksums only.- Hook path remains fail-open to avoid deadlocks/stalls.
Operational recommendations:
- Use workers you control.
- Use dedicated SSH keys for worker access.
- Keep workers patched and isolated.
- Enable telemetry/alerting for production-like use.
Limitations
- Designed around SSH-based Linux worker environments.
- Tooling assumptions are strongest for Rust and selected build/test commands.
- Remote performance gains depend on network + worker capacity + project shape.
- The fleet dashboard (
dashboard/) needs Node and a static host (Vercel/Pages); its live path needs a Vercel Blob store andrchd's[api]listener on a tailnet.
FAQ
Does RCH block my command if the daemon/workers fail?
No. It fails open and allows local execution.
Can I force local or force remote per project?
Yes, via .rch/config.toml (general.force_local / general.force_remote).
Is queue/cancel supported?
Yes. Use rch queue and rch cancel.
Can I inspect why a command is or is not intercepted?
Yes. Use rch diagnose "<command>".
About Contributions
Please don't take this the wrong way, but I do not accept outside contributions for my projects. You can still open issues and PRs for discussion/proof-of-fix, but I review and re-implement changes independently.
License
MIT License with an OpenAI/Anthropic rider — this is not ordinary
OSI MIT: no rights are granted to the restricted parties named in the
rider. See LICENSE for the exact, controlling terms
(SPDX: LicenseRef-MIT-OpenAI-Anthropic-Rider).