stokehold
stokehold runs heavy commands (builds, tests, GPU work) one admission at a time per machine. Several agents share one
laptop; each runs stokehold <cmd> as a background command and is notified when it exits. stokehold decides by itself
whether to start at once or wait in the queue. Nobody polls or checks a status.
stokehold [--cpu N] [--mem SIZE] [--vram SIZE] [--disk SIZE] [--timeout DURATION] [--] cmd args...
stokehold status # for humans: running and queued jobs, reservations, live usage (`--status` works too)
stokehold history [--since 2h|1d|7d] [--html PATH] # how well stokehold used the machine (default: 1d)
The exit code is the child's. Exit 75: the disk is full. A job over its timeout gets SIGTERM to its process group, 10 s of grace, then SIGKILL,
and stokehold exits 124. When the child exits (or is killed), stokehold sends SIGKILL to its whole process group, so
nothing it left behind keeps running or holds a pipe open.
Terminal: when stdin is a terminal and stokehold is in the foreground, the child's group gets the terminal while it
runs (for prompts and pagers) and stokehold takes it back afterwards. If the child stops (Ctrl-Z), stokehold stops too;
fg resumes both.
Weights
Cores, RAM, GPU memory and free disk the job needs. Without flags: the median measured peak (whole process tree,
sampled every 0.5 s while the job runs: peak cores at once and peak RSS) from
~/.local/state/stokehold/history.jsonl for the same command and cwd; else for the same command in the same place of
the same repository (any worktree or clone sharing its git dir, so a fresh agent worktree is not cold); else for the
same program and first argument there (make ui-check, cargo test); else the whole machine's cores and --mem 4G (so cold jobs, Docker commands included, start at least 10 s apart; after one run, history gives the real peak). A request larger than
the machine is clamped, so that job runs alone. On macOS VRAM counts as RAM.
History entries carry a version (v); on Linux, entries from before 0.3.2 (no v) are ignored for defaults because they counted threads as processes.
Admission
All of these must hold:
- VRAM reservations: running VRAM reservations plus this request are at most 90% of capacity. RAM has no reservation sum (see Memory headroom).
- CPU headroom:
90% of all cores - max(instant use, sustained use)leaves one free core. Instant use is non-heavy usage plus the live CPU (or ramp hold) of running jobs; sustained use is the machine's CPU over the last 20 s plus the ramp holds not yet in use. So jobs start while the CPU is at most about 84% busy (83% on 16 cores) and, while jobs wait, the CPU settles in an 80-90% band, never pinned at 100%. The sustained CPU is max(latest, mean of the readings in~/.local/state/stokehold/busy), which every stokehold process adds to (waiters as they re-evaluate, and while any job exists one reading at least every 10 s): a single 250 ms window that falls between build steps can read 80% on a pinned machine, and is not free CPU. A job with a learned (or--cpu) size holds its peak cores until that peak shows in its live use (3/4 of it) or its learned time to peak passes (history records when the CPU first reached 3/4 of its peak; the slowest recent ramp plus 25%, at least 10 s, at most 2 min, 30 s when unknown), the same way a learned memory size is held (a cold job holds the whole machine for 30 s). The unrealized part of a hold reserves the CPU band except a backfill share (2 cores, or 1/8 of the machine): a waiter of that size or smaller still starts beside a held peak while the machine reads idle. So a burst of builds starts one at a time as their peaks show (0.4.3: in 0.4.2 holds ended after 10 s, or after 3 s on a machine reading under 50%, so several 16-core cargo builds started before their CPU showed and the Mac sat at 95-100% CPU with load1 up to 184). Cold jobs (the whole machine for 10 s, so they start at least 10 s apart) and Docker/VM jobs (their CPU never shows in the tree) keep the 10 s ramp hold. A hold is also released once the job's CPU has plateaued (0.4.4): from 10 s after its slowest recent time to peak (at least 20 s after the start), the job's own tree sampler sees the last 10 s flat within itself (highest and lowest differ by at most 1 core, or 1/8 of its level) and marks its job file settled, which wakes the waiters. A learned peak is often one 500 ms spike (make checklearned 15 cores and ran on 1.5), and in 0.4.3 its 2-minute hold kept waiters out at 10-50% CPU. A lull before the slowest ramp never releases, and a settled job whose CPU rises again takes its hold back. Cold,--cpudeclared and Docker/VM jobs are never released by a plateau. Amaketarget that runs containers inside is not detected as Docker/VM work (onlydockerin the command is): its tree CPU reads low, so it may settle while its real work runs in the VM; the measured machine CPU still counts it. A whole-machine (cold) waiter that has waited 60 s (it holds the queue) skips the projected-peak test and starts under the band as in 0.4.3, so one long job cannot leave the machine at 50-60% with jobs queued. - Projected peak (0.4.4): a waiter bigger than the backfill share starts only while half its learned peak fits within every core on top of the projected CPU (the measured use plus the unrealized holds of the jobs still ramping). A 13-core build starts up to about 59% busy on 16 cores, a whole-machine one up to 50%; smaller waiters still fill the band up to one free core. In 0.4.3 any waiter started with one free core under the band, so 13-core builds started at 83% busy and took the Mac to 100% with load1 34. A waiter held back by a hold sleeps until the first moment the holds leave room, and re-samples at least every 10 s, so a peak that shows sooner releases it. A job's recorded peak is the larger of its process tree's peak and the machine-wide CPU rise of other programs over the pre-start baseline, so work hosted in Docker's VM (outside the tree) also teaches a real peak. On Linux a process's threads are not counted as children (sysinfo lists each thread with its whole process's RSS and CPU; before 0.3.2 a 16-thread rustc counted 17 times, so learned memory peaks were inflated about 17x and their reservations blocked idle machines on "mem").
- Memory headroom:
90% of RAM - non-heavy usage - live memory of running jobs - the unused part of each job's reservationfits the request. The unused part (reservation minus live use) of a declared (--mem) size is held for the whole run. A learned size's is held until the job's learned time to peak (history records when memory peaked; 60 s when unknown, at least 10 s), then shrinks to nothing over 60 s; those learned holds together take at most 3/4 of the memory still free. Reservations that merely sum high never block (before 0.4.2 reservations summing over 85% blocked jobs at 43% memory used), but a burst of late-peaking jobs cannot all start either (12 jobs of a 12G peak on a 64G Mac: 3 start, not 12). Docker and WSL VM memory is outside the job's process tree, so for docker jobs the learned peak also counts the machine-wide memory rise over the pre-start baseline, as CPU does. Non-heavy usage is machine-wide use minus the process trees of stokehold's own jobs (Chrome, Discord, other CLIs), smoothed as max(latest sample, mean of the last 3, 10 s apart): a rise blocks at once, a fall is released slowly. On Linux CPU includes steal time (/proc/statfield 8), which WSL2 and VMs otherwise hide. - Known limit: the memory and pressure guards only veto new starts. A gate cannot stop jobs already running, and stokehold does not kill them: if running jobs all grow past their learned peaks, memory can still run out. The unused reservation holds above are what protect against this, not the guards.
- Guards (veto only, they block even a job that would run alone): 1-minute load at or over 4 x cores blocks whatever the CPU says (runaway load: 120 on 16
cores killed Docker on 2026-11-19). From 2 x cores up, load blocks while it is still rising (load1 > 1.1 x load5)
and the CPU is at least 75% busy, or while the CPU is at least 90% busy (the CPU here is the larger of the latest
and the sustained reading). Load without CPU is not CPU overload: on
macOS load also counts threads blocked in I/O (network, disk), so it can read 20-60 on an idle CPU (load1 58 at 26%
CPU during a network outage on 2026-11-20). Memory used below 90%, and no memory
pressure: macOS
kern.memorystatus_vm_pressure_levelof 2 or more, Linux/proc/pressure/memory(some avg1025 or more, orfull avg105 or more; the Windows rig's WSL Ubuntu). An unreadable pressure source never blocks. - Disk: free space on the cwd's filesystem must be at least
--diskplus 10% of the disk. If it is not, waiting cannot help, sostokeholdexits 75 at once and says the disk is full (clean up, or ask Todd). - A job running alone always passes the resource checks. There is no cap on the number of jobs.
Queue: first in, first out. A job behind the head starts when the head and it fit together (it never stops the head from fitting). When the head does not fit now, a job no bigger than the head may backfill past it while the head has waited under 60 s.
State and waiting
~/.local/state/stokehold/jobs/ holds one JSON file per job (pid, process start time, reservation, cmd, cwd, state).
Decisions are made under a flock, which the kernel releases if a process dies. Files of dead jobs are dropped.
Each job's stokehold also holds a flock on its own <id>.lock for its whole life (opened close-on-exec, so a server
the job leaves behind never inherits it). A job whose lock can be taken is dead, however it died (SIGKILL included), and
whatever process now has its pid: its file is deleted and it never keeps a reservation (0.4.3: in 0.4.2 status could
list a running job whose pid no longer existed).
Waiters watch the directory (kqueue/inotify) and re-evaluate when a job file changes or disappears. Only while live
data blocks a job is there a 10 s re-sample (monitoring; a waiter held back by a ramp sleeps until the first moment it fits instead, not until the ramp ends); a 30 s safety sweep catches jobs killed with SIGKILL.
Job files without a lock file (written by 0.4.2 and older) fall back to the pid: dead when the pid is gone, a zombie,
or started at another time than the file says (pid reuse); if the process start time cannot be read, a marker is
stored and liveness checks the pid only. Leftover .tmp files are swept with the dead jobs.
If STOKEHOLD_JOB names a live running job, stokehold execs the command directly.
Stats (stderr)
Queued time (when it matters), run time, CPU user+sys and average cores, peak cores and peak RSS of the whole process tree plus the largest single process, disk blocks,
machine-wide network bytes, peak VRAM and CPU steal where measurable, a TIMED OUT line when it applies, and a
prompt to review the numbers. One JSON line per run is appended to history.jsonl.
History: did stokehold leave the machine idle?
Besides history.jsonl (finished runs, used for default weights), stokehold appends to history-YYYY-MM.jsonl (UTC
months, newest 5 kept) in the state dir. One JSON line per event, one write() on an O_APPEND file, under 4 KB:
queued,started,finished: ts, job id, pid, cwd, repo (git toplevel basename), command (argv[0..2], 80 chars), requested cpu and mem.startedaddswait_msandreasons(time blocked by each of cpu/mem/vram/load1/queue/ramp; older lines may say load5).finishedadds exit code, wall time, cpu time, peak RSS.sample: ts, cpu %, mem %, load1, load5, cores, jobs running, jobs waiting, andblocked(what each waiting job is blocked by right now). While any job waits or runs, every stokehold process checks every 10 s whether a sample is due, and the one that takes the lock writes it, so samples are at most 30 s apart. With no jobs nothing is written (monitoring, exempt from the no-polling rule).
stokehold history prints totals (jobs, total, average and p95 wait), avg cpu while jobs waited (time-weighted;
it should be near full), idle while waiting, a per-repo table (jobs, wait, cpu-hours) and a 5-minute timeline of
cpu, running and waiting. Idle while waiting is the time a job waited although cpu < 75% and memory < 85% (load is
ignored): the time that can be stokehold's fault. It prints
its share of all waiting time and the blockers recorded in those minutes (startable means nothing blocked and the
waiter had not yet woken). --html PATH writes one self-contained page (inline SVG, no scripts) and prints its path.
Configuration
STOKEHOLD_STATE_DIR, STOKEHOLD_TIMEOUT (default 60m). Test overrides: STOKEHOLD_CAPACITY
(cpu=8,mem=16G,vram=0,disk=100G), STOKEHOLD_LIVE (load=,load5=,mem_pct=,mem_pressure=,disk_free=,nonheavy_cpu=,nonheavy_mem=,nonheavy_vram=),
STOKEHOLD_KILL_GRACE_MS.
Renamed from heavy (0.4.0)
The crate and binary were toddstack-heavy / heavy up to 0.3.2. What carries over:
- argv[0] alias: run through a symlink named
heavy(ln -sfn <path>/stokehold ~/.local/bin/heavy) and it behaves identically, so existing sessions and scripts keep working. - State dir:
~/.local/state/stokehold. On the first run an existing~/.local/state/heavyis renamed to it and a symlink is left at the old path, so history is kept and a still-running oldheavybinary follows the link to the samejobs/files and flock locks (one shared queue during the transition). Delete the symlink once no old binary runs. - Env vars:
STOKEHOLD_*are read first, then the legacyHEAVY_*names. Jobs get bothSTOKEHOLD_JOBandHEAVY_JOB, so nesting works across old and new binaries.