Struct SandboxManager 

pub struct SandboxManager { /* private fields */ }

Manages the full lifecycle of multiple sandbox microVMs.

Implementations

impl SandboxManager

pub fn pinned_rootfs_paths(&self) -> Result<BTreeSet<PathBuf>>

Rootfs images that existing snapshots or catalog templates need to stay restorable/bootable.

Exposed so whoever owns the converted-rootfs cache can pin them; see SnapshotCatalog::referenced_rootfs_paths and TemplateCatalog::rootfs_paths. The union matters: a non-prewarmed template pins no snapshot, so without the catalog half a vm-agent update plus a create for the same docker layer would sweep the template’s ext4 as superseded.

pub async fn checkpoint_sandbox( &self, sandbox_id: &SandboxId, name: String, labels: HashMap<String, String>, ) -> Result<CheckpointInfo>

Checkpoint a Ready sandbox into the snapshot catalog.

pub async fn restore_sandbox( &self, spec: RestoreSandboxSpec, ) -> Result<(SandboxId, String)>

Restore a new sandbox from a previously created checkpoint.

The restored sandbox starts in Ready state immediately.

Returns (sandbox_id, ip_address).

pub async fn restore_sandbox_keyed( &self, spec: RestoreSandboxSpec, restore_key: &str, ) -> Result<(SandboxId, String)>

Restore with a stable request key for durable replay.

pub fn list_checkpoints( &self, sandbox_id: Option<&str>, ) -> Result<Vec<CheckpointSummary>>

List checkpoints, optionally filtered by origin sandbox ID.

Internal pause checkpoints are hidden: they are lifecycle state, not user-owned snapshots, and deleting one would strand a paused sandbox. Template-owned snapshots (CORE-107) are hidden for the same reason — they surface via TemplateService.Get/List, not as user checkpoints.

pub async fn delete_checkpoint(&self, snapshot_id: &str) -> Result<()>

Delete a checkpoint by its ID, tearing down any pre-warmed restore slots staged from it first.

Internal pause checkpoints are refused — deleting one would strand its paused sandbox; they die with the sandbox via Remove. Template-owned snapshots (CORE-107) are refused while the catalog still references them — they are reclaimed through template deletion. A labeled snapshot no record references (orphaned by a failed post-commit cleanup) is deliberately deletable: this is the operator’s only recovery path, since list_checkpoints hides it and delete_template answers TemplateNotFound.

impl SandboxManager

pub async fn start_execution( &self, sandbox_id: &SandboxId, spec: ExecutionSpec, ) -> Result<ExecutionSnapshot>

Start an execution inside a Ready sandbox.

The execution id (caller-supplied or generated) addresses the process afterwards: attach, stdin writes, signals, resize, and wait all go through it. Retrying a start with the same id and command returns the existing execution instead of dispatching a second process.

pub fn attach_execution( &self, sandbox_id: &str, execution_id: &str, stdout_offset: u64, stderr_offset: u64, ) -> Result<(ExecutionSnapshot, Receiver<ExecutionOutput>)>

Attach to an execution’s output from the given per-channel offsets.

The receiver yields buffered output first (starting at the earliest retained byte if retention already dropped the requested offset), then live output, and closes once the execution has exited and both channels are drained. Read the final state with SandboxManager::wait_execution afterwards.

pub async fn write_stdin( &self, sandbox_id: &str, execution_id: &str, offset: u64, data: &[u8], eof: bool, ) -> Result<StdinState>

Write stdin bytes at an absolute offset (offset-idempotent).

Bytes below the accepted count are deduplicated so a retried write never double-feeds the process; an offset beyond the accepted count is a gap and is rejected with the resume point. eof closes stdin after the write — rejected for TTY executions, where EOF is a byte (Ctrl-D) the client sends through the stream itself.

pub fn stdin_status( &self, sandbox_id: &str, execution_id: &str, ) -> Result<StdinState>

Current stdin acceptance state — the resume point after a lost write.

pub async fn signal_execution( &self, sandbox_id: &str, execution_id: &str, signal: i32, ) -> Result<()>

Deliver a POSIX signal to a running execution’s process group.

pub async fn resize_execution( &self, sandbox_id: &str, execution_id: &str, width: u16, height: u16, ) -> Result<()>

Resize a running TTY execution’s terminal.

pub async fn wait_execution( &self, sandbox_id: &str, execution_id: &str, timeout: Duration, ) -> Result<ExecutionSnapshot>

Wait until an execution exits or timeout elapses, then return its state. A zero timeout polls the current state.

pub fn list_executions( &self, sandbox_id: &SandboxId, ) -> Result<Vec<ExecutionSnapshot>>

List a sandbox’s retained executions, running and exited, ordered by start time (then id, for a stable order under equal timestamps).

The sandbox must exist; an unknown id is NotFound rather than an empty list, so a caller can tell “no executions” from a typo.

pub async fn wait_sandbox_port( &self, id: &SandboxId, port: u16, timeout: Duration, ) -> Result<()>

Wait until something inside an alive sandbox listens on TCP port, or fail with VmmError::DeadlineExceeded once timeout elapses.

The vm-agent watches the guest’s own listen table in-process — no connect probes that would perturb the workload with spurious accepted connections.

impl SandboxManager

pub async fn read_sandbox_file( &self, id: &SandboxId, path: &str, ) -> Result<Vec<u8>>

Read a file from inside an alive sandbox.

pub async fn write_sandbox_file( &self, id: &SandboxId, path: &str, mode: u32, data: &[u8], ) -> Result<()>

Write a file into an alive sandbox.

pub async fn stat_sandbox_path( &self, id: &SandboxId, path: &str, ) -> Result<FileStatDto>

Stat one path inside an alive sandbox (symlinks reported, not followed).

pub async fn list_sandbox_dir( &self, id: &SandboxId, path: &str, ) -> Result<Vec<FileStatDto>>

List a directory inside an alive sandbox, non-recursively, with full per-entry metadata.

pub async fn make_sandbox_dir( &self, id: &SandboxId, path: &str, mode: u32, ) -> Result<()>

Create a directory (and missing parents) inside an alive sandbox. Succeeds when the directory already exists. mode must already be defaulted by the caller.

pub async fn remove_sandbox_path( &self, id: &SandboxId, path: &str, recursive: bool, ) -> Result<()>

Remove a file, symlink, or directory inside an alive sandbox. A non-empty directory requires recursive.

pub async fn move_sandbox_path( &self, id: &SandboxId, from: &str, to: &str, ) -> Result<()>

Rename / move an entry within an alive sandbox.

pub async fn watch_sandbox_dir( &self, id: &SandboxId, path: &str, recursive: bool, ) -> Result<DirWatch>

Open a directory watch inside an alive sandbox. The returned stream ends cleanly when the sandbox stops; dropping it cancels the watch.

impl SandboxManager

pub async fn replay_sandbox_create( &self, id: &str, request_key: &str, ) -> Result<Option<(SandboxId, String)>>

Replay a durable Create outcome without resolving its template again.

pub async fn create_sandbox( &self, spec: SandboxSpec, ) -> Result<(SandboxId, String)>

pub async fn create_sandbox_keyed( &self, spec: SandboxSpec, request_key: &str, ) -> Result<(SandboxId, String)>

Create a sandbox with a stable key for durable request replay.

pub async fn stop_sandbox( &self, id: &SandboxId, timeout_seconds: u32, ) -> Result<()>

Stop a sandbox gracefully.

Waits up to timeout_seconds (default 30 s) for an active workload to exit, asks the guest to shut down (Ctrl+Alt+Del reboots the guest, which exits the VMM), and the driver SIGKILLs the VMM only if it outlives the remaining budget. All runtime resources (TAP + IP, dm-snapshot CoW, jailer chroot) are released on Stopped; only the inspectable record and the log directory survive until Remove.

pub async fn detach_all(&self) -> Result<()>

Give up ownership of every live VM without stopping it, so this process can exit and the next one adopt them.

The composer calls this on graceful shutdown. Without it a clean exit unwinds into the driver handle’s Drop, which kills — so the startup sweep’s adoption would only ever fire after a crash, never after the binary upgrade or redeploy it mainly exists for.

Best-effort per sandbox: one VM that refuses to be handed over must not cost the others theirs, so failures are collected and reported together. A driver without Detach runs its VMs in-process and has nothing to hand over — the same reasoning the sweep applies to Adopt. The PreparedVm a booted sandbox also holds needs no such step: it kills on drop only while it is still unconsumed.

Each handover goes through its computer’s actor (CORE-145). Detach is a terminal transition, and the actor is what serializes those: taking the handles straight out of the map let a handover run concurrently with the Stop of the same computer — the very stop whose 30s budget is why a composer’s drain deadline expires and this gets called at all. Either order damaged the guest: a stop reaching a detached handle waits out a stood-down reaper and then SIGKILLs the VM it was just told had been handed over, and its release hands TAP, the CoW device and the jail back while the successor is adopting them.

pub async fn remove_sandbox(&self, id: &SandboxId, force: bool) -> Result<()>

Forcibly destroy a sandbox and release all resources immediately.

pub fn inspect_sandbox(&self, id: &SandboxId) -> Result<SandboxInfo>

Return the current state and metadata of a sandbox.

pub fn list_sandboxes( &self, state_filter: Option<&str>, label_filter: &HashMap<String, String>, ) -> Result<Vec<SandboxSummary>>

List sandboxes, optionally filtered by state string and/or labels.

pub fn subscribe_events(&self) -> Receiver<SandboxEvent>

Subscribe to sandbox lifecycle events.

Delivery is best-effort — treat the stream as a latency optimization over polling, never as an authoritative log. Events published before this call are gone (with no subscriber at all they are discarded outright), and a subscriber that falls more than the channel capacity behind gets RecvError::Lagged with the overwritten events lost.

What makes the loss detectable is SandboxEvent::sequence (CORE-147): 1-based, global across all sandboxes of this manager, and contiguous in the order received — so a subscriber that sees sequence jump by more than one has missed events (including any history from before it subscribed) and should fall back to Self::inspect_sandbox / Self::list_sandboxes to re-derive state instead of carrying what it has. That test is conclusive precisely because this subscription is unfiltered; a downstream view that filters it (the wire API’s per-sandbox subscription) sees legitimate gaps and must not read them as loss. The counter is not persisted: a new manager numbers from 1 again, and events emitted while no manager ran were never numbered at all — a consumer that outlives the manager must reconcile on reconnect regardless of sequence.

impl SandboxManager

pub async fn pause_sandbox(&self, id: &SandboxId) -> Result<()>

Pause a Ready sandbox: checkpoint it, then release its runtime resources while keeping the record, checkpoint, and disk overlay under the same id.

Idempotent: pausing a Paused sandbox is a no-op. Any other state answers WrongState — an active execution must finish (or be stopped) first, matching the contract’s “requires READY”.

pub async fn resume_sandbox( &self, id: &SandboxId, resume_reason: &str, ) -> Result<String>

Resume a Paused sandbox in place: restore from its internal pause checkpoint with a fresh network allocation, back to Ready under the same id.

Idempotent: Ready/Running return the current IP without touching anything. reason is surfaced verbatim as the RESUMED event’s “reason” attribute (see reason).

Returns the sandbox’s (fresh) IP address, empty without networking.

impl SandboxManager

pub async fn shutdown(&self)

Release manager-held background resources: tears down every pre-warmed restore slot. Live sandboxes are untouched. On a crash (no shutdown) the startup orphan sweep reclaims slots via their pool-<uuid> crash journals instead.

impl SandboxManager

pub fn get_template(&self, reference: &str) -> Result<ResolvedTemplate>

Resolve a name[:version] catalog reference.

pub fn list_templates(&self) -> Result<Vec<(String, TemplateEntry)>>

Every catalog row: (name, entry) — published versions in publish order, then the draft, grouped by name.

pub async fn register_template_draft( &self, name: &str, entry: TemplateEntry, ) -> Result<TemplateEntry>

Install a built entry as name’s draft, reclaiming any warm snapshot the displaced previous draft owned exclusively.

pub async fn publish_template( &self, name: &str, version: &str, ) -> Result<TemplateEntry>

Freeze name’s draft as immutable version, reclaiming the draft’s exclusively-owned artifacts when an idempotent re-publish collapses a same-digest rebuild.

pub async fn delete_template(&self, reference: &str) -> Result<()>

Delete a template version (name:version) or a whole template (name), including warm snapshots no remaining entry references. Rootfs images are reclaimed separately by the rootfs-cache sweep once Self::pinned_rootfs_paths stops reporting them.

pub async fn promote_snapshot_to_template( &self, source_snapshot_id: &str, template_name: &str, ) -> Result<PromotedSnapshot>

Copy checkpoint source_snapshot_id into a template-owned snapshot (new id, TEMPLATE_LABEL = template_name), decoupling the template’s lifetime from the user checkpoint. Files are reflinked where the filesystem supports it (the data dir is Btrfs), so the copy is cheap; the origin’s rootfs stays pinned through the copied meta.

pub async fn prewarm_template( &self, name: &str, rootfs_path: &str, defaults: &TemplateDefaultsSpec, ) -> Result<PrewarmOutcome>

Boot the built rootfs once with the template’s effective geometry and an empty cmd, checkpoint at READY, and destroy the builder — the snapshot then serves creates as the template’s warm start point. Requires jailer mode (restore does). The builder’s warm behaviour is disabled so a prewarm neither restores from nor publishes into the warm-cache LRU.

pub async fn discard_promoted_snapshot(&self, snapshot_id: &str)

Best-effort removal of a just-promoted copy whose catalog registration failed — without it the copy would sit as a (recoverable) orphan until an operator noticed the log line.

impl SandboxManager

pub async fn set_sandbox_lifecycle( &self, id: &SandboxId, update: LifecycleUpdate, ) -> Result<()>

Replace a computer’s lifecycle deadlines (CORE-60).

ttl_seconds re-arms the hard cap from now (0 removes it); idle_timeout_seconds replaces the idle window, re-arming any live timer; on_idle replaces the policy. None fields are unchanged. Allowed in any non-terminal state — a paused computer keeps honoring its (re-armed) TTL, and new idle knobs apply on the next Ready.

impl SandboxManager

pub async fn run_in_sandbox( &self, id: &SandboxId, cmd: Vec<String>, env: HashMap<String, String>, working_dir: String, user: String, tty: bool, tty_size: Option<(u16, u16)>, timeout_seconds: u32, ) -> Result<Receiver<Result<OutputChunk>>>

impl SandboxManager

pub fn new(config: RuntimeConfig, environment: NodeEnvironment) -> Result<Self>

Create a new manager from the given configuration, over the environment-specific components the composer supplies.

Fails with VmmError::Config when the environment’s driver lacks a capability every sandbox needs — Prepare (the flows spawn the VMM ahead of the guest), Staging (every flow brings its computer’s files into the area the VMM can reach), Vsock (the guest agent is reached over it), or whatever the agent factory’s readiness gate needs (VsockListen for the guest’s dial-out) — or when its guest network offers no NetworkReconcile (the cleanup-token protocol is how a host releases the addresses a previous process held), so an environment missing one is refused here instead of at the first boot or the first cleanup ticket.

pub fn into_shared(self) -> Arc<Self>

Wrap the manager in an Arc and let its computers’ deadline timers fire.

Production embedders (the guest agent’s SandboxService) must use this; a plain Self::new manager never fires idle timers (unit tests exercising unrelated surfaces rely on that inertness).

One watch flip, which every actor is holding a receiver for — the weak self-handle a detached expiry task needed is gone with the detached task.

pub async fn pending_network_cleanups(&self) -> Result<Vec<(String, String)>>

Return sandbox IDs whose inactive network allocation still awaits host forwarding cleanup.

pub async fn validate_network_cleanup( &self, id: &str, token: &str, ) -> Result<NetworkLease>

Validate one exact pending host-cleanup ticket.

Returns the lease that generation held: the host’s forwarding rules for it are keyed by its address, and after an agent restart the quarantine ledger is the only place that address survives.

pub async fn finalize_network_cleanup( &self, id: &str, token: &str, ) -> Result<()>

Recycle one exact inactive sandbox generation after forwarding cleanup.

pub async fn wait_startup_cleanup_complete(&self)

Wait until the current agent generation’s startup cleanup is complete.

pub async fn startup_cleanup_token(&self) -> Result<Option<String>>

Opaque ticket for the host cleanup pass required after agent startup.

pub async fn validate_startup_cleanup(&self, token: &str) -> Result<()>

Validate the current process-generation startup cleanup ticket.

pub async fn finalize_startup_cleanup(&self, token: &str) -> Result<()>

Release the startup gate once host listeners and legacy DNAT are gone.

pub fn sandbox_network_identity( &self, id: &str, ) -> Result<SandboxNetworkIdentity>

Return the active generation’s network identity: the external pool IP (what DNS, expose, and the API report), its opaque cleanup token, and how expose DNAT must target the sandbox (CORE-81/CORE-83).

No startup-cleanup gate here, unlike every method above. An identity needs a live lease; a lease comes only from reserve, which the guest network refuses while the same startup barrier is closed; and both callers in the guest agent await wait_startup_cleanup_complete before asking. The gate this used to take could therefore only ever have fired for a sandbox that has no lease to report anyway, and it has no non-blocking form on the port.