cellos-host-firecracker

The Firecracker microVM [CellBackend] — L2-06 production host backend. Boots one Firecracker VMM per cell behind a jailer and a per-cell manifest, runs spec.run.argv under cellos-init, and reports authenticated exit codes back to the supervisor over a per-cell vsock UDS.

What it is

cellos-host-firecracker implements [cellos_core::ports::CellBackend] on top of a real Firecracker VMM. Each cell is one VMM child process, one chroot under jailer, one ext4 rootfs (read-only when a scratch drive is attached), one writable virtio-blk scratch image (optional), one TAP interface + nftables egress ruleset (Linux only), and one vsock-backed Unix Domain Socket family (_9000 for exit code, _9001 for telemetry).

It is the L2 (host runtime / isolation) production target named in LAYERS.md: "Map an authority bundle to real isolation". The supervisor selects it with CELLOS_CELL_BACKEND=firecracker (crates/cellos-supervisor/src/composition.rs:1031). The backend constructor takes a [FirecrackerConfig] (which can be populated from env via FirecrackerConfig::from_env()); env validation is strict because every flag here trades isolation for ergonomics and we want production misconfigurations to fail closed.

The crate has three sub-modules:

• lib.rs — [FirecrackerConfig], [FirecrackerCellBackend], the CellBackend impl, the in-VM exit-code bridge, the HMAC verification helpers, jailer / chroot wiring, manifest verification, vsock UDS listener (listen_for_exit_code), TAP + nftables setup, scratch ext4 provisioning, graceful-shutdown plumbing, FD-leak guards.
• api_client.rs — minimal hyper client for the Firecracker Management API (PUT /machine-config, PUT /boot-source, PUT /drives/{...}, PUT /network-interfaces/{...}, PUT /vsock, PUT /actions, PUT /snapshot/create, PUT /snapshot/load, PATCH /vm).
• pool.rs — [FirecrackerPool], the warm pool state machine (snapshot → restore) for fast cell startup. The pool is opt-in via CELLOS_FIRECRACKER_POOL_SIZE; default 0 means a zero-slot no-op pool that always misses, so wiring is an inert pass-through on the cold-boot path.

What it deliberately does not do:

• It does not run spec.run.argv as a host subprocess. The argv is base64-encoded into the kernel cmdline as cellos.argv=<b64>; cellos-init reads /proc/cmdline, forks/execs inside the guest, and the supervisor reads the exit code back over vsock. The host-subprocess fallback in cellos-supervisor is shadowed when this backend reports an in-VM exit (src/lib.rs:1-31).
• It does not trust the guest's exit code unauthenticated. The 4-byte little-endian i32 is followed by a 32-byte HMAC-SHA256 tag over exit_code_bytes ‖ cell_id_bytes keyed by a per-cell 32-byte key read from /dev/urandom at create() (FC-18, src/lib.rs:134-218).
• It does not boot non-Linux. The whole backend body is #[cfg(target_os = "linux")]; on Windows/macOS the struct still constructs (so the supervisor composition root type-checks everywhere) but every CellBackend method returns an Unsupported-shaped CellosError::Host (src/lib.rs:632-639).
• It does not terminate TLS in-VM. Per ADR-0004 / ADR-0005, TLS termination lives on the host's egress path, not inside the cell.

Public API surface

FirecrackerConfig (in declaration order, src/lib.rs:238-330):

Architecture / how it works

Lifecycle per cell.

create(spec):

1. Compute the per-cell HMAC key from /dev/urandom (generate_exit_hmac_key, src/lib.rs:161-168).
2. Verify the artifact manifest if manifest_path is set, populating kernel_digest_sha256 / rootfs_digest_sha256 / firecracker_digest_sha256 on the returned handle.
3. Consult the warm pool (pool::checkout); on a hit, PUT /snapshot/load with the snapshot + mem-file paths and skip the PUT /machine-config
   • PUT /boot-source arc. On a miss (or pool disabled), follow the cold-boot path.
4. Stage the chroot (<chroot_base>/firecracker/<cell_id>/root/...), hard-link / copy the rootfs and optional scratch image into the chroot, generate per-cell nftables ruleset from spec.authority.egress, bring up the TAP, spawn firecracker --api-sock <socket> under the jailer (or directly if *_ALLOW_NO_JAILER=1), wait for the API socket to appear (max SOCKET_READY_TIMEOUT = 10s, src/lib.rs:77), drive the Management API.
5. Bind the per-cell exit-code UDS (<vsock_uds_path>_9000) and telemetry UDS (<vsock_uds_path>_9001 — owned by cellos-host-telemetry but rendezvoused in the same socket dir) BEFORE booting the guest.
6. PUT /actions InstanceStart. The guest's cellos-init reads cellos.argv=<base64-json> from /proc/cmdline, forks the workload, then writes a 36-byte authenticated frame (4-byte LE i32 || 32-byte HMAC-SHA256 tag) back to <vsock_uds_path>_9000 before powering off (src/lib.rs:134-148).

wait_for_in_vm_exit(cell_id) reads the 36-byte frame, recomputes the HMAC with the per-cell key, and rejects a mismatched tag in constant time via hmac::Mac::verify_slice (src/lib.rs:172-218).

destroy(handle) sends SendCtrlAltDel via the Management API, waits up to the per-cell graceful-shutdown budget (spec.run.limits.graceful_shutdown_seconds, defaulting to GRACEFUL_SHUTDOWN_TIMEOUT = 5s per src/lib.rs:84 and resolve_graceful_shutdown_timeout at src/lib.rs:93), then SIGKILLs. TAP + nftables table are removed; the chroot is unlinked; the warm-pool slot transitions to Empty and the background filler can re-snapshot.

Warm pool (L2-06-2, src/pool.rs). Cold boot is ~125 ms; restore is ~10 ms. Slot lifecycle is Empty --fill()--> Available --checkout()--> InUse --checkin()--> Empty. checkin returns to Empty (not Available) by design: a VM that ran a cell is no longer at the parked-init snapshot state. A background task re-fills slots after destroy(). The fill task is spawned at supervisor startup when the env var resolves > 0 (crates/cellos-supervisor/src/composition.rs:1051-1075).

HMAC exit auth (FC-18). Without authentication, anything inside the guest with vsock access could spoof a "successful" exit. The 36-byte frame (EXIT_AUTHED_FRAME_LEN, src/lib.rs:147) commits the exit code AND the cell id under a key only the host knows. Both ends use constant-time compare. The verification helper (src/lib.rs:187-218) is pub(crate); tests reach it through a doc-hidden __fc18 shim.

Manifest verification (FC-08). When CELLOS_FIRECRACKER_MANIFEST points to a v1 manifest file, create() re-hashes the kernel, rootfs, and Firecracker binary before boot. Digest mismatch is a hard error. The opt-out is intentionally two flags — see "Configuration" below.

Jailer. The jailer drops to jailer_uid / jailer_gid (10002 by default) and chroots into <chroot_base>/firecracker/<cell_id>/root. The API socket then lives at <chroot_base>/firecracker/<cell_id>/root/run/firecracker.socket (src/lib.rs:244-247). The require-jailer flag is on by default and flipping it off requires the explicit *_ALLOW_NO_JAILER=1 opt-out that emits a loud warning (src/lib.rs:268-269).

Non-Linux stub. Outside Linux, the CellBackend impl returns CellosError::Host from every method; tests can still link the crate on macOS / Windows hosts (src/lib.rs:1334-...).

Configuration

All env vars are read by FirecrackerConfig::from_env() (or the from_lookup helper for tests). Required absolute paths fail-closed at init if absent.

The two-flag manifest opt-out exists because a single env var can be set in a base image, a Helm chart copied between environments, or an .env file leaking from dev to prod by mistake. Requiring a paired _REALLY=1 forces the operator to make the trade-off explicit on the same line, in the same operation (src/lib.rs:280-285).

Supported guest architectures

The backend boots two guest architectures, matching the host architecture (the guest kernel and rootfs must be built for the same arch as the host that runs Firecracker):

The Rust backend itself is arch-agnostic — FirecrackerCellBackend boots whatever vmlinux + rootfs CELLOS_FIRECRACKER_KERNEL_IMAGE and CELLOS_FIRECRACKER_ROOTFS_IMAGE point at; the manifest digests cover both arches identically. The arch-specific surface lives in the build pipeline (scripts/firecracker/); see that directory's README for build invocations and the per-arch CONFIG_* exemption list.

Apple Silicon developer note: Firecracker requires KVM, which Apple Silicon's macOS userspace does not expose. To run the FC backend on a Mac, run it inside a Linux/arm64 VM (Lima, OrbStack, UTM) whose guest kernel exposes /dev/kvm. The build scripts and the backend itself work identically there. The host stub backend (cellos-host-stub) remains the right choice for native-macOS development without a Linux VM.

Examples

use std::sync::Arc;
use cellos_core::ports::CellBackend;
use cellos_host_firecracker::{FirecrackerCellBackend, FirecrackerConfig};

// From env (validates required paths up front; fails fast on misconfig).
let backend: Arc<dyn CellBackend> = Arc::new(FirecrackerCellBackend::from_env()?);

// Or pin the config inline.
let cfg = FirecrackerConfig::from_env()?;
let backend: Arc<dyn CellBackend> = Arc::new(FirecrackerCellBackend::new(cfg));

// Optional: attach an EventSink so `cell.firecracker.v1.pool_checkout`
// CloudEvents flow alongside the regular lifecycle stream.
// let backend = FirecrackerCellBackend::from_env()?.with_event_sink(sink);
# Ok::<(), cellos_core::CellosError>(())

Warm-pool sizing:

export CELLOS_FIRECRACKER_POOL_SIZE=8

# Supervisor spawns one background fill task at startup.

# pool::pool_size_from_env() resolves the value at backend construction.

Testing

# Unit tests (always run, no KVM required).
cargo test -p cellos-host-firecracker

# Integration suite — requires Linux + /dev/kvm + opt-in.
cargo test -p cellos-host-firecracker -- --ignored

The tests/ directory contains 41 integration tests. The #[ignore]-gated subset needs Linux, /dev/kvm, the jailer binary, CAP_NET_ADMIN for nftables / TAP manipulation, and (in some cases) the artifact manifest. Highlights:

The cleanup_on_crash.rs file panics with a stable FC-37 GAP: marker on the reconcile_orphans slots so a future implementer can lift the #[ignore] without re-discovering the contract.

A small subset (e.g. host_capabilities_smoke.rs) runs on every CI leg without --ignored.

Related crates

• cellos-core — the CellBackend trait, CellHandle, TeardownReport, EgressRule, ExecutionCellDocument, ExecutionCellSpec, CellosError.
• cellos-init — the in-guest PID-1 that reads cellos.argv=<b64> from /proc/cmdline, forks the workload, computes the HMAC tag, and writes the 36-byte frame back over vsock.
• cellos-host-telemetry — pairs the _9000 exit-code UDS this crate owns with the _9001 telemetry UDS in the same socket dir.
• cellos-host-stub — no-op backend used in unit tests of the supervisor pipeline.
• cellos-host-gvisor — runsc alternative for hosts without KVM.
• cellos-supervisor — selects this backend with CELLOS_CELL_BACKEND=firecracker and spawns the warm-pool fill task.

ADRs

• ADR-0001 — Rust + NATS/JetStream + proprietary host — the surrounding stack commitments that frame Firecracker as one isolation primitive among several.
• ADR-0004 — TLS termination fronting / trust boundary — TLS termination is an egress-edge concern, not in-VM, so this backend does not interpose on TLS.
• ADR-0005 — TLS termination design — the design that backend authors must respect: egress goes through declared rules; this crate translates them into nftables.
• ADR-0006 — In-VM observability runner evidence — the receiver side (cellos-host-telemetry) is the host half of the observability system this backend ships the channel for.