Zygo
Warm sandboxes for function-shaped code — daemonless, rootless, OCI-compatible.
This page is the first page of the Zygo book. Read it for the idea in five minutes; the book takes it from there.
Zygo runs webhook handlers, agent tools, cron jobs and data transforms with
Docker's ergonomics, but without the container create/destroy cycle. The sandbox
waits warm; a request costs a fork().
Why
A warm process that serves many requests is fast and dirty: request n sees
whatever request n-1 left — a monkeypatch, a cached connection, a mutated
module, an atexit handler. A container per request is clean and slow:
300–1000 ms, of which your code is 5–20 ms.
Zygo is the third thing. zygo serve starts an interpreter, lets it do its
imports, and parks it. zygo exec forks it. The child is a copy of a process
that has never served a request, so it is as clean as a fresh container
and as cheap as a fork — a median of 1.4 ms against 300–1000.
docker exec |
a shared worker process | zygo exec |
|
|---|---|---|---|
| Overhead per request | 50–100 ms | ~0 | 1.4 ms usually (1 in 100: 10.5 ms on Linux 6.x, 2.6 ms on 5.10) |
| What request n can see of n-1 | everything | everything | nothing |
| Limits per request | the container's | none | its own cgroup: memory, pids, CPU, a deadline |
| A request that overruns | kills the container | kills the worker | killed through its own cgroup; the zygote keeps serving |
| Paid once, up front | a docker run -d |
your worker's start | a zygo serve: ~150 ms for a Python handler, plus your imports |
WARM ── pay once, then request after request
═══════════════════════════════════════════════════════════════════════════
docker run -d ──▶ one container, shared state
│
├─ docker exec ▶ dockerd ▶ containerd ▶ shim ▶ runc ▶ process
├─ docker exec ▶ dockerd ▶ containerd ▶ shim ▶ runc ▶ process
│ same state, every time
▼
docker rm 50–100 ms per exec
zygo serve ──▶ warm zygote: interpreter up, imports done, waiting
│
├─ zygo exec ▶ fork() ▶ process
├─ zygo exec ▶ fork() ▶ process
│ clean copy, every time
▼
zygo down ~1.7 ms per exec
A handler is a function — def handler(event) — and everything around it is
the runtime's: the fork, the per-request cgroup, the deadline, the secrets
written outside the sandbox and removed afterwards, and the metrics. A
compiled program needs none of that and gets the same treatment through
warm-exec: the sandbox is held and each request is a fresh process running
your cmd, at about 2 ms. There is a protocol
(spec/protocol.md) rather than an interface, so an agent
in any language gets all of it — Zygo ships one for Python and one for Node,
and zygo agent test checks anything else against the same conversation.
And the one-shot case, underneath
Everything above is built on an ordinary sandbox, and that sandbox is worth having by itself:
Running a 30-line Python function in a container costs 300–1000 ms, and that overhead is not isolation — a namespace set costs about 1 ms, a cgroup 0.1 ms, a seccomp filter microseconds. It is orchestration: daemon → containerd → shim → runc, and a container object left behind to remove.
docker run |
docker exec |
zygo run |
zygo exec (warm) |
|
|---|---|---|---|---|
| Overhead per request | 300–1000 ms | 50–100 ms | 12 ms | 1.4 ms |
| What that pays for | daemon, shim, runc, a container object |
the daemon round trip | namespaces, cgroup, mounts — in one process | a fork() |
| Paid once, up front | — | a docker run -d: 300–1000 ms |
— | a zygo serve: ~150 ms for a Python handler, plus your imports |
| Daemon | yes | yes | no | no |
| Root | yes | yes | no | no |
| Clean state per request | yes | no | yes | yes |
| Boundary | kernel | kernel | kernel, or gVisor | kernel |
Zygo's two numbers are medians with the image cached, on the machines named in
what Zygo costs — a Raspberry Pi 5 and two
VMs on an Apple-silicon Mac, all aarch64; Docker's are its commonly measured
range. The program's own start-up is on top of every column. zygo bench all
reproduces every one of them on your host, prints the machine it ran on, and
refuses to give a verdict if that machine was throttled or busy.
ONE-SHOT ── one request, one fresh sandbox
════════════════════════════════════════════════════════════════════
docker run zygo run
────────── ────────
docker CLI zygo
│ │ clone3 · mounts · cgroup
▼ │ seccomp · Landlock · execve
dockerd ▼
│ your program
▼ │
containerd ▼ exit
│ nothing left behind
▼
shim
│
▼
runc
│
▼
your program
│
▼ exit
container object stays → docker rm
300–1000 ms ~12 ms
Try it
Not released yet. The commands below are for the first release, v0.1.0, and start working when it is tagged. Until then, build from a checkout:
cargo build --release -p zygo-cligivestarget/release/zygo.
# Linux, x86_64 or aarch64: one static binary, no runtime dependencies
|
# macOS: the shim, the Linux build it forwards into, and Lima
# or from source, anywhere with Rust
# or the container image, which needs no privileges and three specific things
Every release lists the archives' checksums in SHA256SUMS, and the container
image is signed with cosign;
chapter 11 shows how to check both.
packaging/oci/ says what those three are and why; zygo doctor names any that are missing, in the container or on a host.
# the warm path, which is the point
# and the sandbox underneath it
Linux needs kernel 5.3 or newer, unprivileged user namespaces and cgroup v2
controllers delegated to your user; 6.1 or newer is recommended, because that
is where Landlock's network rules, cgroup.kill and memory.peak are all
present. zygo doctor attempts each requirement rather than reading a setting,
and prints the fix for anything missing. On Ubuntu and Debian two AppArmor
policies get in the way of sandboxes and of networked sandboxes respectively;
troubleshooting has both, and doctor names them.
macOS gets a Linux VM. Every sandbox command is forwarded into one that
Zygo starts and manages, with the same arguments, working directory and
streams, and your home directory mounted at the same path. It needs limactl,
which starts the VM, and a Linux build of Zygo to put inside it. The Homebrew
formula installs both; from a checkout they are:
Crossing into the VM costs about 22 ms per command once it is up — the shim
uses the SSH connection Lima already holds — so a one-shot run from a Mac
shell is about 29 ms, of which ~6 ms is the sandbox. The millisecond warm path
is there through the API and the SDKs, and through zygo api running inside
the VM. Getting started has the details, and
what Zygo costs has the numbers.
How
Three ideas:
- A sandbox is a constrained process, not a container. Namespaces, cgroups, seccomp and Landlock, set up in one process with no RPC.
- The sandbox waits warm; nothing is built on the request path. Compiled binaries spawn into a ready sandbox in 1–3 ms (warm-exec). Where interpreter start is expensive, a small in-sandbox agent warms it once and forks per request — copy-on-write, so no copying, but no leaked state either.
- Isolation is one flag.
ns(namespaces),gvisor(userspace kernel),vm(libkrun). Same spec, same command, same protocol.
What is in the box
One file describes a project:
[]
= "python:3.12-slim"
= "256M"
= 0.5
= "30s"
= "none"
[]
= "./resize.py" # defines handler(event); warmed once, forked per request
= "./requirements.txt"
= ["libwebp7"] # apt packages, installed once as a layer
= "512M"
[]
= "alpine:3" # no runtime → warm-exec: a fresh process per request
= ["./bin/parse:/app/parse:ro"] # a static binary you built, mounted in
= ["/app/parse"] # stdin: JSON event, stdout: JSON result
[]
= "./fetch.py"
= "egress" # nothing else is reachable
= ["api.stripe.com:443", "*.example.com:443"]
= ["STRIPE_KEY"] # delivered as a file, only to the request's process
- Every limit is mandatory — memory, CPU, pids, wall clock, scratch, open files — with a default, and no way to disable one without a flag you have to type. The deadline kills the request's whole process tree.
- Networking is off by default.
egressis an allowlist by name, enforced by nftables inside the sandbox's own namespace with a resolver Zygo controls; private ranges and the cloud metadata address stay refused. No privilege anywhere:pastamoves the packets as your own user. - Dependencies never touch the image.
requirementsbecomes a venv built with the image's ownpip;systembecomes an OCI layer of its own. Both are built once and shared by everything that names the same thing. zygo upis a deploy, not a restart. Unchanged functions are left warm; changed ones are replaced blue/green. It writeszygo.lockwith the digest each image resolved to, and refuses to run a moved image silently.- Secrets are files, for the duration of a request. Read from your shell, written by the supervisor from outside the sandbox at mode 0400, never in the environment and never in the warm agent's memory.
- Any language. Python handlers get the fork path. Anything else is
warm-exec with a
cmd, or an agent of your own — the protocol is language independent, andzygo agent testchecks an implementation against it.
From a program, the same functions are behind an HTTP API with bearer auth and two dependency-free clients:
= # a unix socket, or 127.0.0.1:7700
= . # ~2 ms, a fresh process
import from 'zygo-sdk';
const out = .;
For an agent host, zygo mcp speaks the Model Context Protocol over a pipe.
The tools expose a program and nothing else — no image, no mounts, no network,
no limits — because a model reads untrusted text and that text can ask it for
things. Those are set once, by whoever installed the server:
Status
Measured, not asserted. On a Linux 6.8 VM, the warm path is a median of
1.44 ms through the shipping code at 250 requests a second — with a 99th
percentile of 10.5 ms, a kernel cgroup cost the book explains — sustaining
1,108 requests a second at a concurrency of four; a cold zygo run with
the image cached is a median of 12.3 ms. The warm path is the
production shape, and the gap is the argument: the book
shows a multi-tenant consumer — one warm zygote per script version — on it. The suites run in three
places, which turned out to matter: a privileged container, a Raspberry Pi as
an ordinary user under a systemd session, and a Mac. The launcher is checked
against a real kernel, including actual escape attempts; every syscall number
the architecture has is swept against all three seccomp profiles; and fifty
scenarios shaped by use case rather than by mechanism run on two of the three.
What Zygo costs has the numbers, the hosts, and what is
not measured — and zygo bench all reproduces every one of them on your own
host, printing the machine it ran on and refusing to give a verdict when that
machine was throttled or busy.
Scoped, not unfinished. The vm backend boots a guest and runs one-shot
sandboxes — about 420 ms against ns's 73 ms on the same host, for a kernel
of the guest's own. The guest can write, to a private layer bounded by
scratch and never to the shared image. It has no network and no warm
functions, and gvisor has neither either; both refuse them with a reason
rather than weakening something. That is a decision rather than a gap —
ADR 0002 says why, and what would
reopen it. Warm functions are an ns feature.
Zygo scales to one machine, and answers 429 past its capacity. No external
audit has been done.
Documentation
Everything is in the Zygo book — one book, in plain English, with diagrams throughout. It starts from zero and ends with the full reference.
| Part I — Container 101 | The kernel, namespaces, cgroups, seccomp and Landlock, and Docker: what a sandbox is made of. |
| Part II — Zygo, explained | How Zygo works, where it saves, its principles, FreeBSD jails, and every similar project — docker run against zygo run, flag by flag. |
| Part III — Using Zygo | Getting started, one-shot sandboxes, warm functions, limits and networking, images, production, the API and SDKs, writing an agent. |
| Part IV — Reference | Every command, every sandbox.toml field, environment, files and exit codes, troubleshooting. |
| Part V — Security and speed | The threat model, seccomp profiles, and what Zygo costs. |
| Part VI — Decisions | Why it is built this way, and the design records. |
examples/ |
A webhook, a CI job, an LLM tool, a Go program, and agents in Node and POSIX sh. |
spec/protocol.md |
The wire protocol between the supervisor and an agent. |
Layout
crates/zygo-core the library; the CLI and the bindings sit on top
spec/ sandbox.toml surface, layering, validation
image/ OCI references, content-addressed store, registry client
sandbox/ mount plan and resource limits, backend independent
cgroup.rs the two-level cgroup v2 hierarchy
backend/ ns | gvisor | vm
protocol/ the warm execution wire protocol
doctor.rs environment probing
crates/zygo-cli the `zygo` binary
agents/python the reference Python agent and its conformance suite
agents/node the reference Node agent: a worker pool, not a fork
spec/protocol.md the wire protocol
sdk/python the Python client, and the async one beside it
sdk/node the Node client, with types and no build step
packaging/oci the container image, and a worker image built on it
Development
Four rules the test suite is built on, all learned the hard way here:
- A test must attempt the thing, not inspect a setting. Reading a flag passes on a kernel that ignores the flag. Every escape case runs the escape.
- A test must not disturb what it measures. Checking
isatty(1)through a pipe, or comparing a sandbox's terminal while redirecting stdout, measures the pipe. - A latency measurement must be able to say whether it hit a limit. A
closed-loop benchmark under a hard CPU quota measures the quota:
zygo bench warmreads the tenant'scpu.statand declines to judge the p99 budget when the tenant was throttled, because that number would be about the limit rather than about this code. - A negative check must first prove the thing ran. "The connection was refused", "no process leaked" and "the suite exited zero" are all satisfied by nothing having happened at all — the failure mode that fails open. Each one establishes the positive case first.
Several checks here passed — or failed — for the wrong reason before those rules were applied.
Security
Zygo runs other people's code on purpose, so an escape is the most serious kind of bug it can have. SECURITY.md is how to report one — privately, through GitHub, not as an issue — and what is in scope. docs/book/23-security.md lists every vector, the control against it, and whether the escape suite actually attempts it. It also has a section on where the boundary is weaker than it looks, which is the part worth reading before you trust this with anything.
No external audit has been done.
Licence
Apache-2.0.
Next: The Zygo booklet →