arcbox-computer-runtime

Guest-side sandbox orchestration: runs inside the ArcBox System VM and manages nested microVMs — one per sandbox — through a VmDriver (arcbox-vm-driver); the reference driver is arcbox-fc-driver, Firecracker.

Do not confuse it with arcbox-vmm, which is the host VMM that boots the System VM itself on Virtualization.framework or KVM. Different layer, different machine.

host (macOS)         System VM (Linux)              sandbox (microVM)
arcbox-daemon  ──vsock──►  arcbox-agent        ──vsock──►  vm-agent (PID 1)
  arcbox-vmm             arcbox-computer-runtime             workload
                         arcbox-vm-driver                (arcbox-vm-agent)
                         (arcbox-fc-driver)
                                    └── arcbox-vm-proto ──┘

arcbox-agent owns the sandbox.v1 surface and the vsock transport, and calls ComputerManager underneath. There are no service implementations, no tonic, and no daemon here. Node image builders can use RootfsBuilder independently.

The vm-agent binary that becomes PID 1 inside each sandbox is a separate crate, arcbox-vm-agent; the wire vocabulary both sides share — boot parameters, exec-channel and file-channel frames — is arcbox-vm-proto. This crate and the agent each depend on the proto crate and never on each other, so the agent stays a small static musl binary no matter what the manager pulls in. boot_proto and file_proto stay reachable here as re-exports. RootfsBuilder (this crate) stages the binary into every sandbox rootfs at /sbin/vm-agent — OCI/overlay2 → ext4 conversion plus the default busybox image, with the agent binary source, the cache directory, and the busybox supplied by the composer as RootfsPaths. The rootfs convention the boot protocol relies on is therefore implemented once, here.

RootfsBuilder::inject_vm_agent(&image) also injects the configured agent into an existing ext4 image. Injection requires Linux and loop-mount privileges. Resolver setup and resource cleanup failures return an error. An owned task completes injection and cleanup after caller cancellation. Converted images keep their distribution's /sbin/init; RuntimeConfig selects VM_AGENT_PATH with the default init= boot argument. Custom boot arguments must select the same path.

RootfsBuilder::build_rootfs(RootfsSpec { source, out, size }) builds an image at a caller-owned path and capacity. RootfsSource accepts an OCI layout, an overlay2 directory, or a resolved oci2rootfs::ImageSource. Capacity must be a positive multiple of ROOTFS_CAPACITY_GRANULARITY (128 MiB). Both the declared ext4 capacity and file length must equal the requested capacity; sources that make the formatter enlarge the image fail before agent injection. The builder injects the agent before atomically replacing the destination with a synchronous rename. Failed or cancelled builds preserve an existing destination. On cancellation, an owned task finishes conversion and injection before removing the temporary image. Cleanup failures return an error, or are logged when the caller has gone.

convert_layer_to_rootfs starts cached templates at 512 MiB and lets the formatter grow the image to fit the source. Templates share agent injection, cancellation cleanup, and atomic publication with build_rootfs. Only build_rootfs enforces an exact caller-supplied capacity. Cache names remain rootfs-<layer>-<agent>.ext4; capacity does not change template identity.

These cancellation guarantees require the Tokio runtime to keep running. Runtime shutdown or process termination can leave temporary images, mount directories, or loop devices behind. The default busybox rootfs builder uses a separate pipeline.

Enable remote-image to resolve registry references with the re-exported oci2rootfs::RemoteRef. Keep the resolved source's config and manifest digest before passing RootfsSource::Image(source) to build_rootfs. Use fully qualified references: the current dependency misparses bare numeric tags such as debian:12. The opt-in registry integration test defaults to docker.io/library/debian:12, builds a 32 GiB sparse image, and reads the result through the Linux ext4 driver:

sudo -E env ARCBOX_REQUIRE_BLOCK_TOOLS=1 cargo test -p arcbox-computer-runtime --features remote-image --test integration \
  a_registry_image_builds_at_sparse_computer_capacity -- --ignored --nocapture

Usage

use std::sync::Arc;
use arcbox_computer_runtime::{NodeEnvironment, RuntimeConfig, ComputerManager, ComputerSpec};

// `environment` is the composer's; see below.
let manager = Arc::new(ComputerManager::new(RuntimeConfig::default(), environment)?);

let (id, ip) = manager
    .create_computer(ComputerSpec {
        vcpus: 1,
        memory_mib: 512,
        ..Default::default()
    })
    .await?;

RuntimeConfig loads from TOML (RuntimeConfig::from_file) or is built programmatically; [firecracker], [network], and [defaults] are the sections that matter, and [firecracker.jailer] opts into jailer mode. See config/runtime.rs for the fields.

[firecracker] is named for the VMM a deployed vmm.toml has always run, not for anything this crate knows: the keys that configure a VMM adapter live in the same section and are read by whoever builds it (for the System VM, arcbox_agent::config::AdapterConfig). serde ignores unknown fields, so one section serves both halves.

What the stack needs from its environment — the pieces that differ between the System VM's busybox userland and a stock distro — is a separate input, NodeEnvironment, and all four members are required. This crate builds none of them and names no VMM: whoever composes the node does. For the System VM that is arcbox_agent::sandbox::node_environment.

driver — the VMM behind arcbox_vm_driver::VmDriver. It must claim Prepare and Staging and offer vsock, or ComputerManager::new refuses it.
network — what the NICs attach to, behind arcbox_vm_driver::net::GuestNetwork. It must offer NetworkReconcile.
agent — how the guest agent is reached, behind agent::GuestAgentFactory. The reference is the arcbox-vm-proto client over the driver's vsock, which also decides what the readiness gate needs from the driver.
cow_manager — the copy-on-write rootfs manager (arcbox_snapshot::snapshot_cow::CowManager), built over the composer's own loop-device tooling: BusyboxBlockTools in the System VM, UtilLinuxBlockTools on a stock distro, or a consumer's own few dozen lines of ioctls.

testkit::fake_environment (feature testkit) is those four over the port's fakes, for tests that must not need a VMM.

The sandbox network itself — the IPv4 pool, the per-sandbox TAP, the invariant NAT (eBPF TCX or netfilter), and the quarantine ledger — is arcbox-tap-net, the Linux adapter of the arcbox-vm-driver GuestNetwork port; the composer builds it, and this crate does not depend on it at all. Rendering the invariant translation in the host's netfilter framework is that adapter's own seam (arcbox_tap_net::PacketFilter: IptablesLegacy in the System VM, Nftables on a stock distro — legacy and nft rulesets are mutually invisible), so it is an argument to the network, not a member here.

Build and test

cargo test -p arcbox-computer-runtime    # unit + integration, no Firecracker needed
cargo clippy -p arcbox-computer-runtime -- -D warnings
cargo test -p arcbox-tap-net             # the TAP network (root-only TAP tests skip)

# vm-agent, as the release ships it (aarch64 musl; brew install FiloSottile/musl-cross/musl-cross)
cargo build -p arcbox-vm-agent --bin vm-agent --target aarch64-unknown-linux-musl --release

tests/manager_over_fakes.rs drives the manager's own flows — create, boot, exec, files, pause, resume, checkpoint, restore, expiry, remove, adoption — over the driver port's FakeDriver/FakeNetwork and this crate's FakeAgent, so the whole lifecycle is testable with no KVM, no root and no Firecracker.

Everything real — TAP creation (arcbox-tap-net), boot, checkpoint — needs Linux with CAP_NET_ADMIN, a firecracker binary, and (for jailer mode) root, and so belongs to whoever composes the driver and the network. That is arcbox-agent: its tests/sandbox_manager_e2e.rs and examples/sandbox-manager-smoke.rs boot real Firecracker through the environment its SandboxService composes. .github/workflows/test-vm-linux.yml runs the whole ladder on real KVM.

Jailer mode: every path the FC API sees is chroot-relative

Firecracker pivot_roots before it processes any API request, so a host-absolute path handed to the API resolves inside the chroot and returns ENOENT. Kernel and rootfs are therefore staged into the chroot before boot ({chroot}/vmlinux, {chroot}/rootfs.ext4) and the API is given /vmlinux and /rootfs.ext4. Checkpoints write to {chroot}/snapshots/{id}/ and are moved out to the catalog afterwards; restore copies them back into the new sandbox's chroot first.

The same rule is what keeps vsock sockets from colliding: the vmstate records the UDS as /run/firecracker.vsock, which is a different host path in every sandbox's chroot.

Durable sandbox records retain kernel and rootfs paths so an adopted sandbox can checkpoint, pause, and resume. The manager pins each record's source rootfs until the record is removed. Snapshots and templates also pin their source rootfs. Workload inputs and SSH keys are redacted when provisioning completes.

Legacy records without kernel or rootfs paths remain adoptable. Their checkpoint requests return FailedPrecondition before capture and leave the guest running.

Restore inherits the source snapshot's kernel, rootfs, vCPU count, and memory size. Snapshots without recorded geometry, or with a zero vCPU count or memory size, return FailedPrecondition before reserving resources. Re-checkpoint the source sandbox with the current agent to create a restorable snapshot.

Data layout

{data_dir}/                       # default /var/lib/firecracker-vmm
├── kernels/vmlinux
├── images/*.ext4
├── sandboxes/{id}/               # firecracker.sock, .vsock, .log, .metrics
├── sandbox-network-quarantine/   # arcbox-tap-net's ledger: {id}.json per quarantined address
└── snapshots/{sandbox-id}/{snapshot-id}/
                                  # vmstate, mem, meta.json

Under jailer mode the live files instead sit beneath {chroot_base_dir}/firecracker/{sandbox-id}/root/ (default /srv/jailer), and the snapshot catalog above is still the durable home.

License