Skip to main content

microvms_core/
sandbox.rs

1// SPDX-License-Identifier: Apache-2.0
2//! The `Sandbox` lifecycle: one VM's whole life, with the state machine the symspec
3//! model describes made true of the code.
4//!
5//! Composes [`crate::control`] and [`crate::session`] into the surface the CLI and the
6//! bindings use: build, run, suspend, resume, terminate, the launch-time suspended
7//! window (STATE-12), and teardown in the order that does not leak.
8//!
9//! # The lifecycle is a field, and every transition is guarded
10//!
11//! [`Lifecycle`] is the symspec's `vm_state` verbatim, and [`Sandbox`] carries the other
12//! four variables beside it — `token_installed`, `image_exists`, `was_terminated`,
13//! `bootstrap_count`. The Z3 proofs over that model (bootstrap at most once, suspend from
14//! non-RUNNING unreachable, TERMINATED never returns to RUNNING) are proofs about *this*
15//! struct's reachable states, which is only worth something if the transitions here are
16//! the only way to move it. They are: every one of those fields is private, and every
17//! mutation happens in one of the five methods below.
18//!
19//! # Runtime-checked rather than typestate, deliberately
20//!
21//! The packet offered `Sandbox<Running>` returning a `Suspended` handle, which would make
22//! STATE-5's wrong call a compile error rather than a local refusal — strictly stronger on
23//! the ladder in [`crate`]. It is not what landed, for the reason the packet names as
24//! acceptable: T-W3-8 wraps **one** object for PyO3 and napi-rs, and a type whose Rust
25//! identity changes on every transition cannot be one `#[pyclass]`. A typestate sandbox
26//! would be re-erased into a runtime-checked enum at the binding boundary, so the check
27//! would exist twice with the binding's copy being the one most callers actually hit.
28//!
29//! What is kept from the typestate idea is the part that costs nothing: the check happens
30//! **before** the wire call, so a suspend from SUSPENDED is refused with zero
31//! control-plane calls rather than answered by AWS. The test asserts the call count, which
32//! is the observable that distinguishes the two.
33//!
34//! # The suspended window is checked before the call (STATE-12)
35//!
36//! The launch-time `idlePolicy` *terminates* a suspended VM once `suspendedDurationSeconds`
37//! passes, which means "resume later" silently stops working. A resume past the window is
38//! refused locally, before `ResumeMicrovm`, because the alternative is calling and reading
39//! the failure: the service answers about a terminated id, which is not the same statement
40//! as "the window you set at launch closed", and getting there costs the full poll timeout
41//! first. The window comes from this sandbox's own `RunMicrovm` request, falling back to the
42//! `idlePolicy` that `GetMicrovm` reports (measured 2026-08-15, `docs/PLATFORM.md`). Only
43//! the suspend's start time is the client's alone.
44//!
45//! # Teardown never raises, and the log group is last
46//!
47//! [`Sandbox::terminate`] returns a [`TeardownReport`] rather than a `Result`. It runs
48//! where a caller's `finally` would, and an error raised there replaces the real failure
49//! with a teardown failure — the real one being the one worth reading.
50//!
51//! The order is VM, then image (retrying), then the log group **last**, because the
52//! service can recreate a group deleted before its image. See
53//! [`TeardownReport::undeleted`] for what this crate can and cannot delete.
54//!
55//! # There is no `Drop` that tears down
56//!
57//! Rust has no context manager and `Drop` cannot await. A `Drop` that blocked on a runtime
58//! would deadlock inside one; a `Drop` that spawned would race the process exit. So
59//! [`Sandbox`]'s `Drop` only **warns** about a live VM, naming the id, and the rule is
60//! that a caller calls [`Sandbox::terminate`] explicitly.
61
62use std::sync::Arc;
63use std::time::Duration;
64
65use crate::control::{
66    ControlPlane, CreateImageRequest, Image, Microvm, RunHookPayload, RunMicrovmRequest, WaitOpts,
67};
68use crate::error::{Error, ErrorKind};
69use crate::region::Region;
70use crate::session::{Session, TokenMinter};
71
72/// The default launch wait: five minutes, matching the Python client's `ready_timeout_sec`.
73pub const DEFAULT_READY_TIMEOUT: Duration = Duration::from_secs(300);
74
75/// The default lifecycle wait for suspend, resume, and terminate.
76pub const DEFAULT_LIFECYCLE_TIMEOUT: Duration = Duration::from_secs(300);
77
78/// How many times a teardown retries the image delete.
79///
80/// Twenty, from the Python client: an image in `CREATING` refuses deletion and a VM still
81/// terminating holds a reference, so this is the difference between a clean account and a
82/// billed leak rather than politeness.
83pub const DEFAULT_DELETE_ATTEMPTS: u32 = 20;
84
85/// The gap between image-delete attempts.
86pub const DEFAULT_DELETE_BACKOFF: Duration = Duration::from_secs(15);
87
88/// Where the advisory deny points a well-behaved client: a loopback port nothing serves.
89///
90/// Loopback rather than an unroutable public address, because a refused connection is
91/// immediate and a black-holed one costs the workload a connect timeout per request. Port 1
92/// is privileged, so a demoted workload cannot answer it by accident.
93pub const DENY_EGRESS_PROXY_URL: &str = "http://127.0.0.1:1";
94
95/// The environment variables the advisory deny sets, in both spellings clients read.
96///
97/// Both cases on purpose: `curl` and most Unix clients read the lowercase names, Python's
98/// `requests`/`urllib3`, npm and the AWS SDKs read the uppercase ones, and a client that
99/// reads only the spelling this list omits is a hole in a mechanism that is advisory to
100/// begin with. `no_proxy` is deliberately absent: the launch environment starts empty
101/// (`agentd` clears it and applies only what the hook delivered), so there is no inherited
102/// exemption to overwrite, and each key spends payload budget.
103pub const DENY_EGRESS_ENV_KEYS: [&str; 6] = [
104    "http_proxy",
105    "https_proxy",
106    "all_proxy",
107    "HTTP_PROXY",
108    "HTTPS_PROXY",
109    "ALL_PROXY",
110];
111
112/// How often a lifecycle wait polls.
113const LIFECYCLE_POLL_INTERVAL: Duration = Duration::from_secs(5);
114
115/// The symspec's `vm_state`, verbatim.
116///
117/// Six states and no others, which is the S1 half of this module: a lifecycle held as a
118/// `String` would let `"RUNNING "` and `"Running"` both exist, and every guard below would
119/// have to decide which it meant.
120#[derive(Clone, Copy, Debug, Eq, PartialEq)]
121pub enum Lifecycle {
122    /// The initial state, and the state a launch is accepted into (STATE-1).
123    Pending,
124    /// The run hook answered with a success status and the token is installed (STATE-2).
125    Running,
126    /// A suspend was accepted and the platform has not yet reported it complete (STATE-4).
127    Suspending,
128    /// The platform reported suspension complete (STATE-6).
129    Suspended,
130    /// A terminate was accepted (STATE-9).
131    Terminating,
132    /// The platform reported termination complete (STATE-10).
133    Terminated,
134}
135
136impl Lifecycle {
137    /// The name the service uses for this state, which is also what an error message says.
138    pub fn as_str(self) -> &'static str {
139        match self {
140            Lifecycle::Pending => "PENDING",
141            Lifecycle::Running => "RUNNING",
142            Lifecycle::Suspending => "SUSPENDING",
143            Lifecycle::Suspended => "SUSPENDED",
144            Lifecycle::Terminating => "TERMINATING",
145            Lifecycle::Terminated => "TERMINATED",
146        }
147    }
148
149    /// The lifecycle the service's `state` names, or `None` for a spelling this client does
150    /// not know. `constants::MICROVM_STATES` is the closed set, and a test ties the two.
151    pub fn from_service(state: &str) -> Option<Self> {
152        Some(match state {
153            "PENDING" => Lifecycle::Pending,
154            "RUNNING" => Lifecycle::Running,
155            "SUSPENDING" => Lifecycle::Suspending,
156            "SUSPENDED" => Lifecycle::Suspended,
157            "TERMINATING" => Lifecycle::Terminating,
158            "TERMINATED" => Lifecycle::Terminated,
159            _ => return None,
160        })
161    }
162
163    /// Whether a VM in this state is still billing, which is what a `Drop` warning is for.
164    pub fn is_live(self) -> bool {
165        matches!(
166            self,
167            Lifecycle::Pending | Lifecycle::Running | Lifecycle::Suspending | Lifecycle::Suspended
168        )
169    }
170}
171
172impl std::fmt::Display for Lifecycle {
173    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
174        f.write_str(self.as_str())
175    }
176}
177
178/// Everything a launch needs, with the defaults the Python client measured.
179///
180/// `agent_token` is optional because the common case is a per-VM secret nobody needs to
181/// see; a caller who has one already — a harness minting its own, or a retry that must
182/// reuse the first attempt's — passes it.
183#[derive(Clone, Debug)]
184pub struct RunRequest {
185    /// The image to launch, or `None` for the one [`Sandbox::build_image`] built.
186    pub image_identifier: Option<String>,
187    /// `imageVersion`, or `None` for the image's own latest active version.
188    ///
189    /// Pinning is what makes a canary a canary: without it a launch takes whatever version is
190    /// latest at the moment the call lands, which is not necessarily the one the caller just
191    /// built. Paired with [`crate::control::ControlPlane::set_image_version_status`] it is also
192    /// the rollback — set the bad version INACTIVE, re-pin here to the good one — and a version
193    /// set INACTIVE refuses to launch when named here.
194    ///
195    /// See [`crate::control::RunMicrovmRequest::image_version`], which this forwards to.
196    pub image_version: Option<String>,
197    /// The execution role. Optional in the model; every real launch needs one.
198    pub execution_role_arn: Option<String>,
199    /// The bearer token the daemon will accept, or `None` to mint one.
200    pub agent_token: Option<String>,
201    /// Persist this unique launch key before sending a request; reuse only for that same
202    /// launch and identical parameters. None mints a fresh key. A stable key requires
203    /// an explicit agent_token and identity=false so retries carry identical payloads.
204    pub client_token: Option<String>,
205    /// Base environment for every exec in the launched VM, delivered in the same
206    /// `runHookPayload` as the token.
207    ///
208    /// The daemon applies this *under* each request's own `env`, so a per-exec value
209    /// wins on a key both set. Empty by default, and an empty map produces byte-for-byte
210    /// the payload this client always sent — a caller who never touches this field
211    /// cannot be affected by the field existing.
212    ///
213    /// It shares the token's 4096-byte payload budget, and the check is local:
214    /// [`crate::control::RunHookPayload::for_launch`] refuses an over-ceiling payload
215    /// before any call, naming the byte count and how much of it the env is. That
216    /// matters here more than for the token, because one bearer token has always fit
217    /// with room to spare and a map of credentials does not.
218    pub launch_env: std::collections::HashMap<String, String>,
219    /// Whether to generate and deliver a tunnel identity (#70 layer 3).
220    ///
221    /// On, the launch generates two fresh x25519 seeds, delivers the VM's seed and the
222    /// host's public key in the same payload as the token, and [`RunOutcome`]-equivalent
223    /// state on the session keeps the [`crate::identity::TunnelIdentity`] — the host secret
224    /// and the VM's public pin — for `tunnel --verify-identity` to prove the far end.
225    /// Off (the default) sends byte-for-byte the payload this client always sent.
226    ///
227    /// A flag rather than always-on, deliberately: the material spends 137 bytes of the
228    /// payload's measured 4096-byte budget, and a caller near the ceiling with a launch env
229    /// deserves to choose which feature gets the room.
230    pub identity: bool,
231    /// Whether to request the egress connector. Off omits it from the request; measured
232    /// 2026-09-12 the platform gave a connector-less VM outbound network anyway
233    /// (`docs/PLATFORM.md`).
234    pub egress: bool,
235    /// Customer-managed Lambda VPC egress connector ARNs. See
236    /// [`RunMicrovmRequest::egress_network_connectors`].
237    pub egress_network_connectors: Vec<String>,
238    /// Sets advisory proxy-deny environment variables for clients that honor them.
239    /// Workloads can bypass these variables. Internet isolation requires a VPC egress
240    /// connector using a VPC without an internet gateway or NAT gateway.
241    ///
242    /// Refused with [`RunRequest::egress`]. Existing caller-supplied proxy variables win;
243    /// the added variables share the 4096-byte run-hook payload budget.
244    pub deny_egress: bool,
245    /// Whether to launch shell-capable: the ingress set becomes the measured pair
246    /// `[HTTP_INGRESS, SHELL_INGRESS]` instead of `ALL_INGRESS`, which is what
247    /// `CreateMicrovmShellAuthToken` requires and what `microvm shell` attaches to.
248    /// Off (the default) launches byte-for-byte what this client always sent.
249    pub shell: bool,
250    /// `idlePolicy.maxIdleDurationSeconds`.
251    pub max_idle_sec: u32,
252    /// `idlePolicy.suspendedDurationSeconds` — the window STATE-12 refuses past.
253    pub suspended_sec: u32,
254    /// `idlePolicy.autoResumeEnabled`.
255    pub auto_resume: bool,
256    /// `maximumDurationInSeconds`, checked against 1..=28800 before the call.
257    pub max_duration_sec: u32,
258    /// How long to wait for RUNNING.
259    pub ready_timeout: Duration,
260    /// A label for the run token (TRAP-1). Never the token.
261    pub token_scope: Option<String>,
262    /// Per-VM `logging` (#201). See [`RunMicrovmRequest::logging`].
263    pub logging: Option<crate::control::ops::Logging>,
264    /// Whether [`Sandbox::run`] waits for RUNNING before returning (the default).
265    ///
266    /// Off returns as soon as `RunMicrovm` is accepted, with the lifecycle still PENDING:
267    /// the session is addressable, and [`Sandbox::wait_until_running`] finishes the wait
268    /// later — from this process, or after a durable workflow's next step.
269    pub wait: bool,
270}
271
272impl Default for RunRequest {
273    fn default() -> Self {
274        Self {
275            image_identifier: None,
276            image_version: None,
277            execution_role_arn: None,
278            agent_token: None,
279            client_token: None,
280            launch_env: std::collections::HashMap::new(),
281            identity: false,
282            egress: false,
283            egress_network_connectors: Vec::new(),
284            deny_egress: false,
285            shell: false,
286            max_idle_sec: 600,
287            suspended_sec: 600,
288            auto_resume: false,
289            max_duration_sec: 3_600,
290            ready_timeout: DEFAULT_READY_TIMEOUT,
291            token_scope: None,
292            logging: None,
293            wait: true,
294        }
295    }
296}
297
298impl RunRequest {
299    /// A launch with the measured defaults: ingress only, ten-minute idle and suspended
300    /// windows, a one-hour maximum duration, no auto-resume.
301    pub fn new() -> Self {
302        Self::default()
303    }
304
305    /// Launches `identifier` rather than the built image.
306    #[must_use]
307    pub fn with_image(mut self, identifier: impl Into<String>) -> Self {
308        self.image_identifier = Some(identifier.into());
309        self
310    }
311
312    /// Requests the managed internet egress connector. Omission does not block internet access.
313    #[must_use]
314    pub fn with_egress(mut self) -> Self {
315        self.egress = true;
316        self
317    }
318
319    /// Routes egress through a customer-managed Lambda network connector.
320    #[must_use]
321    pub fn with_egress_network_connector(mut self, arn: impl Into<String>) -> Self {
322        self.egress_network_connectors.push(arn.into());
323        self
324    }
325
326    /// Replaces managed internet egress with customer-managed VPC connectors, when any are
327    /// given; an empty list leaves the request unchanged.
328    ///
329    /// The two cannot be combined, so a launch surface that requests managed egress by
330    /// default (an agent VM) uses this to switch rather than failing the launch.
331    #[must_use]
332    pub fn with_vpc_egress(mut self, arns: Vec<String>) -> Self {
333        if !arns.is_empty() {
334            self.egress = false;
335            self.egress_network_connectors = arns;
336        }
337        self
338    }
339
340    /// Applies the advisory in-guest deny. See [`RunRequest::deny_egress`] for what it is
341    /// and, more importantly, for what it is not.
342    #[must_use]
343    pub fn with_deny_egress(mut self) -> Self {
344        self.deny_egress = true;
345        self
346    }
347
348    /// What this launch's outbound network **is**, as opposed to what it asked for.
349    ///
350    /// The one derivation, so the CLI's envelope, the CLI's human line and any embedding
351    /// consumer cannot disagree about the same launch.
352    pub fn egress_posture(&self) -> crate::control::EgressPosture {
353        crate::control::EgressPosture::for_launch(self.egress, self.deny_egress)
354    }
355
356    /// The launch environment as the guest will see it: the caller's, plus the advisory
357    /// proxy deny when [`RunRequest::deny_egress`] is on.
358    ///
359    /// The caller's own value wins on a shared key. A launch that already routes through a
360    /// proxy of its own has said where its traffic goes, and overwriting that with a black
361    /// hole would break a working configuration in the name of a posture this client
362    /// already declines to call a seal.
363    pub fn effective_launch_env(&self) -> std::collections::HashMap<String, String> {
364        let mut env = self.launch_env.clone();
365        if self.deny_egress {
366            for key in DENY_EGRESS_ENV_KEYS {
367                env.entry(key.to_string())
368                    .or_insert_with(|| DENY_EGRESS_PROXY_URL.to_string());
369            }
370        }
371        env
372    }
373
374    /// Requests a shell-capable launch. See [`RunRequest::shell`].
375    #[must_use]
376    pub fn with_shell(mut self) -> Self {
377        self.shell = true;
378        self
379    }
380
381    /// Pins the launch to one `imageVersion`. See the field for why.
382    #[must_use]
383    pub fn with_image_version(mut self, version: impl Into<String>) -> Self {
384        self.image_version = Some(version.into());
385        self
386    }
387
388    /// Sets the suspended window this sandbox will refuse a resume past (STATE-12).
389    #[must_use]
390    pub fn with_suspended_sec(mut self, seconds: u32) -> Self {
391        self.suspended_sec = seconds;
392        self
393    }
394
395    /// Reuses a caller-supplied agent token rather than minting one.
396    #[must_use]
397    pub fn with_agent_token(mut self, token: impl Into<String>) -> Self {
398        self.agent_token = Some(token.into());
399        self
400    }
401
402    /// Generates and delivers a tunnel identity with the launch (#70 layer 3).
403    ///
404    /// After a `run` with this set, [`Sandbox::tunnel_identity`] holds what
405    /// `tunnel --verify-identity` needs: the host's secret and the VM's public pin.
406    #[must_use]
407    pub fn with_identity(mut self) -> Self {
408        self.identity = true;
409        self
410    }
411
412    /// Adds one launch-environment variable, which every exec in the VM starts with.
413    ///
414    /// One pair per call rather than a whole map, because that is how a caller builds
415    /// one — from flags, from a config file, one credential at a time — and a
416    /// map-taking setter makes the second call silently discard the first. The field is
417    /// public for a caller who really does hold a map.
418    #[must_use]
419    pub fn with_launch_env(mut self, key: impl Into<String>, value: impl Into<String>) -> Self {
420        self.launch_env.insert(key.into(), value.into());
421        self
422    }
423}
424
425/// What a teardown should delete beyond the VM itself.
426///
427/// Both deletions are opt-in, because both destroy something a caller may still want: the
428/// image is reusable across runs, and the log group is where a failed build's only evidence
429/// lives.
430#[derive(Clone, Copy, Debug)]
431pub struct TeardownOpts {
432    /// Whether to delete the image the sandbox built.
433    pub delete_image: bool,
434    /// Whether the build log group should be deleted.
435    ///
436    /// This crate **cannot** delete it — CloudWatch is not in the dependency set — so
437    /// asking names the group in [`TeardownReport::undeleted`] rather than removing it.
438    /// See that field for why naming is the honest answer rather than a silent success.
439    pub delete_log_group: bool,
440    /// How many times the image delete is retried.
441    pub delete_attempts: u32,
442    /// The gap between image-delete attempts.
443    pub delete_backoff: Duration,
444    /// How long to wait for TERMINATED, or `None` to return as soon as the terminate call
445    /// is accepted.
446    ///
447    /// `None` by default, matching the Python client: the caller is on the way out, and a
448    /// teardown that blocked five minutes on a state nobody reads is five minutes of a CI
449    /// job. A caller that needs STATE-10 *observed* passes
450    /// [`TeardownOpts::waiting_for_terminated`].
451    pub wait_for_terminated: Option<Duration>,
452}
453
454impl Default for TeardownOpts {
455    fn default() -> Self {
456        Self {
457            delete_image: false,
458            delete_log_group: false,
459            delete_attempts: DEFAULT_DELETE_ATTEMPTS,
460            delete_backoff: DEFAULT_DELETE_BACKOFF,
461            wait_for_terminated: None,
462        }
463    }
464}
465
466impl TeardownOpts {
467    /// Deletes the image as well as the VM.
468    #[must_use]
469    pub fn deleting_image(mut self) -> Self {
470        self.delete_image = true;
471        self
472    }
473
474    /// Asks for the log group too, which names it rather than deleting it.
475    #[must_use]
476    pub fn deleting_log_group(mut self) -> Self {
477        self.delete_log_group = true;
478        self
479    }
480
481    /// Waits for TERMINATED before returning (STATE-10).
482    #[must_use]
483    pub fn waiting_for_terminated(mut self) -> Self {
484        self.wait_for_terminated = Some(DEFAULT_LIFECYCLE_TIMEOUT);
485        self
486    }
487}
488
489/// What a teardown did, and what it left behind.
490///
491/// Returned rather than raised. See the module docs: this runs where a `finally` would.
492#[derive(Clone, Debug, Default, Eq, PartialEq)]
493pub struct TeardownReport {
494    /// Identifiers of everything a caller asked to have deleted that still exists.
495    ///
496    /// The CLI emits these (CLI-6), which is the whole reason they are identifiers rather
497    /// than a boolean: a leak nobody can name is a leak nobody can clean up.
498    ///
499    /// Two things land here. A delete that was attempted and failed — an image whose twenty
500    /// attempts all hit a conflict. And the build **log group**, which this crate cannot
501    /// delete at all: no CloudWatch client exists anywhere in this workspace, and a logs
502    /// API for one delete call is a bigger surface than the leak it would hide (an
503    /// `aws-sdk-cloudwatchlogs` edge is a reasonable future add if teardown grows more
504    /// log-group work). Naming it is strictly better than the
505    /// alternatives: deleting it is unavailable, and silently succeeding would report a
506    /// clean teardown over six accumulated log groups — which is how the leak was found in
507    /// the first place.
508    pub undeleted: Vec<String>,
509    /// Whether the terminate call was accepted.
510    pub terminate_accepted: bool,
511    /// Whether the image was deleted, or `None` when deletion was not asked for.
512    pub image_deleted: Option<bool>,
513    /// The lifecycle state the sandbox ended in.
514    pub lifecycle: Option<Lifecycle>,
515    /// Every failure the teardown swallowed, in the order it hit them.
516    ///
517    /// Kept because a teardown that never raises is a teardown whose failures are invisible
518    /// otherwise, and the first one is usually the cause of the rest.
519    pub failures: Vec<String>,
520}
521
522impl TeardownReport {
523    /// Whether anything a caller asked for was left behind.
524    pub fn leaked(&self) -> bool {
525        !self.undeleted.is_empty()
526    }
527}
528
529/// Mints proxy tokens for one MicroVM through the control plane.
530///
531/// The bridge between the two lanes: `ControlPlane::mint_auth_token` answers a
532/// `control::ProxyToken`, [`TokenMinter`] wants a `session::ProxyToken`, and the `From`
533/// impl at `session/proxy.rs` is the conversion — one `.into()` at the boundary, which is
534/// why that impl is `From` rather than a named function.
535///
536/// The plane sits behind an `Arc` rather than a reference because the minter must outlive
537/// the call that built the session: minting happens inside the request path on every later
538/// request, which is what makes it happen at all (TRAP-9).
539pub(crate) struct ControlPlaneMinter {
540    pub(crate) control: Arc<ControlPlane>,
541    pub(crate) microvm_id: String,
542}
543
544impl TokenMinter for ControlPlaneMinter {
545    fn mint(
546        &self,
547    ) -> futures_util::future::BoxFuture<'_, Result<crate::session::ProxyToken, Error>> {
548        Box::pin(async move {
549            let minted = self.control.mint_auth_token(&self.microvm_id).await?;
550            Ok(minted.into())
551        })
552    }
553
554    /// Overrides the default, which ignores the ports and delegates.
555    ///
556    /// This is the minter that has a control plane behind it, so it is the one that can
557    /// actually widen a token's scope — and without this override
558    /// `Session::connect_headers(8080)` would keep answering a header pair behind a token
559    /// scoped to 9000 only, which the proxy refuses with 403 `Access to port denied`. See
560    /// [`TokenMinter::mint_for_ports`] for the measurement.
561    fn mint_for_ports(
562        &self,
563        ports: &[u16],
564    ) -> futures_util::future::BoxFuture<'_, Result<crate::session::ProxyToken, Error>> {
565        let specs: Vec<crate::control::ops::PortSpecification> = ports
566            .iter()
567            .map(|port| crate::control::ops::PortSpecification::port(*port))
568            .collect();
569        Box::pin(async move {
570            let minted = self
571                .control
572                .mint_auth_token_for(&self.microvm_id, &specs)
573                .await?;
574            Ok(minted.into())
575        })
576    }
577}
578
579/// One MicroVM's whole life.
580///
581/// See the module docs for the state machine, the window, and why teardown is explicit.
582pub struct Sandbox {
583    control: Arc<ControlPlane>,
584    image: Option<Image>,
585    microvm: Option<Microvm>,
586    session: Option<Session>,
587    /// A backend for the session [`Sandbox::run`] builds, or `None` for the real HTTP one.
588    ///
589    /// The daemon-side sibling of `with_control_plane`'s transport seam: a test that
590    /// scripts the control plane can launch, but the session `run` then builds would dial
591    /// a real endpoint. This lets the same test script the daemon too. Never set in
592    /// production — `Session::builder` picks reqwest when no backend is given.
593    session_backend: Option<crate::session::SharedBackend>,
594
595    // ── the symspec's five variables ─────────────────────────────────────────
596    lifecycle: Lifecycle,
597    /// Every transition of `lifecycle`, for a reader that must not take the caller's lock
598    /// (a keepalive task running while the same sandbox is busy inside a long exec).
599    lifecycle_watch: tokio::sync::watch::Sender<Lifecycle>,
600    token_installed: bool,
601    image_exists: bool,
602    was_terminated: bool,
603    bootstrap_count: u32,
604
605    /// The window from *our own* `RunMicrovm` request. `GetMicrovm` also reports it (in
606    /// `idlePolicy`), which is the fallback when this sandbox has no request of its own.
607    suspended_window: Option<Duration>,
608    /// Whether the accepted launch carried a caller client token, and so may have adopted
609    /// a VM an earlier attempt launched (#195).
610    launch_adoptable: bool,
611    /// The egress posture of this sandbox's own accepted launch, or `None` for a sandbox
612    /// that launched nothing (an adopted one never saw the launch options).
613    launch_posture: Option<crate::control::EgressPosture>,
614    /// The launch's agent token, held until the session that carries it is built. Kept out
615    /// of `Debug` like every other credential here.
616    pending_agent_token: Option<String>,
617    /// `maxIdleDurationSeconds` from our own `RunMicrovm` request: a keepalive's cadence
618    /// is checked against it.
619    idle_window: Option<Duration>,
620    /// The clock reading when the suspend call was accepted, or `None` when not suspended.
621    suspended_at: Option<Duration>,
622    /// Set by [`Sandbox::terminate`], so `Drop` can tell an abandoned VM from a torn-down
623    /// one.
624    torn_down: bool,
625    /// Built by [`Sandbox::adopt`] around a VM another process launched. That process owns
626    /// the VM's teardown, so dropping this handle is not a leak and `Drop` stays quiet.
627    adopted: bool,
628    /// Set by [`Sandbox::detach`]: the VM was handed to another process, which now owns its
629    /// lifecycle and teardown. Every transition is refused and `Drop` stays quiet.
630    detached: bool,
631    /// The launch's tunnel identity, when [`RunRequest::identity`] asked for one.
632    ///
633    /// Holds the host secret and the VM's *public* pin — the VM's own secret was dropped
634    /// the moment the payload was built ([`crate::identity::LaunchIdentity::keep`]), so
635    /// nothing on the host side can impersonate the VM it launched.
636    tunnel_identity: Option<crate::identity::TunnelIdentity>,
637    /// The STS and S3 calls [`Sandbox::ensure_image`] makes, or `None` until the first
638    /// call builds the real ones. The seam a test replaces them at.
639    build_services: Option<Arc<dyn crate::control::BuildServices>>,
640    /// The caller's account, resolved once per sandbox for image ARNs (IMAGE-8).
641    account: Option<String>,
642}
643
644impl std::fmt::Debug for Sandbox {
645    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
646        // No agent token: it is a credential, and a `Debug` that printed it would put it in
647        // every log line that formats a sandbox.
648        f.debug_struct("Sandbox")
649            .field("lifecycle", &self.lifecycle)
650            .field("microvm", &self.microvm.as_ref().map(|vm| &vm.id))
651            .field("image", &self.image.as_ref().map(|image| &image.identifier))
652            .field("token_installed", &self.token_installed)
653            .field("bootstrap_count", &self.bootstrap_count)
654            .field("was_terminated", &self.was_terminated)
655            .field("suspended_window", &self.suspended_window)
656            .field("adopted", &self.adopted)
657            .field("detached", &self.detached)
658            .finish_non_exhaustive()
659    }
660}
661
662/// What another process needs to adopt a VM this sandbox handed off with
663/// [`Sandbox::detach`]: pass the fields to [`Sandbox::adopt`] (or [`Sandbox::adopt_in`]).
664///
665/// The agent token is a credential: readable through [`Detached::agent_token`] so it can be
666/// persisted, and absent from `Debug`, so a log line that formats the record cannot leak it.
667/// Store it the way [`crate::names::NameRecord`] asks: privately, encrypted.
668#[derive(Clone)]
669pub struct Detached {
670    /// The VM's identifier.
671    pub microvm_id: String,
672    /// The HTTPS endpoint its daemon answers on.
673    pub endpoint: String,
674    /// The region the VM runs in.
675    pub region: Region,
676    /// The daemon port the endpoint's proxy tokens are minted for.
677    pub port: u16,
678    agent_token: String,
679}
680
681impl Detached {
682    /// The bearer the VM's daemon accepts; required by [`Sandbox::adopt`].
683    pub fn agent_token(&self) -> &str {
684        &self.agent_token
685    }
686}
687
688impl std::fmt::Debug for Detached {
689    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
690        f.debug_struct("Detached")
691            .field("microvm_id", &self.microvm_id)
692            .field("endpoint", &self.endpoint)
693            .field("region", &self.region)
694            .field("port", &self.port)
695            .field("agent_token", &"<redacted>")
696            .finish()
697    }
698}
699
700impl Sandbox {
701    /// Resolves credentials for `region` and returns a sandbox with nothing launched.
702    ///
703    /// TRAP-6 is closed by [`Region`] before this is reached, which is why there is no
704    /// region check here.
705    pub async fn new(region: Region) -> Result<Self, Error> {
706        Ok(Self::with_control_plane(ControlPlane::new(region).await?))
707    }
708
709    /// A sandbox over an already-built [`ControlPlane`].
710    ///
711    /// The seam every test uses, and public for the same reason `ControlPlane::with_transport`
712    /// is: a caller who wants a non-default port, or a fake transport, has already built the
713    /// plane, and a sandbox constructible only from a [`Region`] would be a sandbox no test
714    /// can drive without AWS.
715    ///
716    /// The plane's own clock is what times the suspended window, rather than a second one
717    /// this type would own. Two clocks in one lifecycle is the trap the daemon's turmoil
718    /// tests record: a window measured on one clock while the poll loop sleeps on another
719    /// calls a closed window open.
720    pub fn with_control_plane(control: ControlPlane) -> Self {
721        Self {
722            control: Arc::new(control),
723            image: None,
724            microvm: None,
725            session: None,
726            session_backend: None,
727            lifecycle: Lifecycle::Pending,
728            lifecycle_watch: tokio::sync::watch::Sender::new(Lifecycle::Pending),
729            token_installed: false,
730            image_exists: false,
731            was_terminated: false,
732            bootstrap_count: 0,
733            suspended_window: None,
734            launch_adoptable: false,
735            launch_posture: None,
736            pending_agent_token: None,
737            idle_window: None,
738            suspended_at: None,
739            torn_down: false,
740            adopted: false,
741            detached: false,
742            tunnel_identity: None,
743            build_services: None,
744            account: None,
745        }
746    }
747
748    /// Routes [`Sandbox::ensure_image`]'s STS and S3 calls through `services`.
749    ///
750    /// The seam beside `with_control_plane`'s transport: a test that scripts the control
751    /// plane replaces the two calls outside it here, and production builds
752    /// [`crate::control::SignedBuildServices`] on the first `ensure_image`.
753    pub fn with_build_services(mut self, services: Arc<dyn crate::control::BuildServices>) -> Self {
754        self.build_services = Some(services);
755        self
756    }
757
758    /// Routes the session [`Sandbox::run`] builds through `backend` instead of real HTTP.
759    ///
760    /// The daemon-side half of the test seam `with_control_plane` opens: scripting the
761    /// control plane gets a test through the launch, and this gets it through everything
762    /// the launched session then does — uploads, execs, downloads — without a daemon.
763    pub fn with_session_backend(mut self, backend: crate::session::SharedBackend) -> Self {
764        self.session_backend = Some(backend);
765        self
766    }
767
768    // ── the symspec's five variables, readable ───────────────────────────────
769
770    /// The lifecycle state, which is the symspec's `vm_state`.
771    pub fn lifecycle(&self) -> Lifecycle {
772        self.lifecycle
773    }
774
775    /// Whether the agent token has been installed (STATE-2).
776    pub fn token_installed(&self) -> bool {
777        self.token_installed
778    }
779
780    /// Whether an image is recorded as existing (STATE-1).
781    pub fn image_exists(&self) -> bool {
782        self.image_exists
783    }
784
785    /// Whether this VM was ever terminated (STATE-11).
786    pub fn was_terminated(&self) -> bool {
787        self.was_terminated
788    }
789
790    /// How many times the token has been installed. Never above one (STATE-3).
791    pub fn bootstrap_count(&self) -> u32 {
792        self.bootstrap_count
793    }
794
795    /// The image, once built.
796    pub fn image(&self) -> Option<&Image> {
797        self.image.as_ref()
798    }
799
800    /// The VM as the service last described it.
801    pub fn microvm(&self) -> Option<&Microvm> {
802        self.microvm.as_ref()
803    }
804
805    /// The session, once launched.
806    pub fn session(&self) -> Option<&Session> {
807        self.session.as_ref()
808    }
809
810    /// The suspended window this sandbox asked for at launch, once it has launched.
811    pub fn suspended_window(&self) -> Option<Duration> {
812        self.suspended_window
813    }
814
815    /// A receiver that sees every lifecycle transition without taking this sandbox's lock.
816    ///
817    /// A binding's keepalive reads it between polls: a sandbox busy inside a long exec
818    /// holds the lock the whole time, which is exactly when the keepalive must keep polling,
819    /// and a suspend or terminate must end the keepalive before its next poll auto-resumes
820    /// the VM.
821    pub fn watch_lifecycle(&self) -> tokio::sync::watch::Receiver<Lifecycle> {
822        self.lifecycle_watch.subscribe()
823    }
824
825    fn set_lifecycle(&mut self, lifecycle: Lifecycle) {
826        self.lifecycle = lifecycle;
827        self.lifecycle_watch.send_replace(lifecycle);
828    }
829
830    /// The idle window this sandbox asked for at launch, once it has launched.
831    pub fn idle_window(&self) -> Option<Duration> {
832        self.idle_window
833    }
834
835    /// The tunnel identity, when the launch asked for one ([`RunRequest::identity`]).
836    ///
837    /// What `tunnel --verify-identity` verifies with: the host's secret and the VM's public
838    /// pin. `None` on a launch that did not ask, and the tunnel route then refuses an
839    /// identity request with close code 4401 rather than downgrading.
840    pub fn tunnel_identity(&self) -> Option<&crate::identity::TunnelIdentity> {
841        self.tunnel_identity.as_ref()
842    }
843
844    /// The egress posture of this sandbox's launch (BIND-12), which its session carries.
845    ///
846    /// The launch's own [`RunRequest::egress_posture`] once [`Sandbox::run`] was accepted.
847    /// Otherwise [`EgressPosture::default`], `unsealed`: an adopted sandbox never saw the
848    /// launch options, and the weakest true claim is the only one it can make.
849    ///
850    /// [`EgressPosture::default`]: crate::control::EgressPosture::default
851    pub fn egress_posture(&self) -> crate::control::EgressPosture {
852        self.launch_posture.unwrap_or_default()
853    }
854
855    /// The agent port the control plane was built with: the hooks port on every image
856    /// this sandbox builds, and the port the daemon listens on in every VM it launches.
857    /// Read-only; `agents::AgentVm` needs it to derive a Dockerfile whose `AGENTD_PORT`
858    /// agrees with the launch.
859    pub fn port(&self) -> u16 {
860        self.control.port()
861    }
862
863    // ── build ────────────────────────────────────────────────────────────────
864
865    /// Builds an image and waits for it to become usable.
866    ///
867    /// Every local guard runs inside [`ControlPlane::create_image`], before the call —
868    /// which matters because the create happens *after* the caller's artifact upload, so a
869    /// rejection AWS raises costs the upload first.
870    pub async fn build_image(&mut self, request: CreateImageRequest) -> Result<&Image, Error> {
871        let size = request.size;
872        let created = self.control.create_image(request).await?;
873        let mut built = self
874            .control
875            .wait_for_image(&created.identifier, size, WaitOpts::default())
876            .await?;
877        // The logging config survives the wait from the create's own record, because the
878        // readback cannot carry it: `GetMicrovmImage` reports no logging member, and the
879        // resolved log stream's discriminator was minted inside `create_image` — this is
880        // the only copy, and dropping it here would leave the caller unable to name the
881        // stream their build wrote to.
882        built.log_group = created.log_group;
883        built.log_stream = created.log_stream;
884        // Recorded before any launch, because a built image exists whether or not anything
885        // is ever run from it — and the teardown has to be able to name it either way.
886        self.image_exists = true;
887        self.image = Some(built);
888        Ok(self.image.as_ref().expect("just assigned"))
889    }
890
891    /// Builds or reuses the content-addressed image for `request` (#221). Not yet
892    /// implemented.
893    pub async fn ensure_image(
894        &mut self,
895        request: crate::control::EnsureImageRequest,
896    ) -> Result<crate::control::EnsuredImage, Error> {
897        // Everything local first: a request this client refuses costs no call at all, the
898        // caller-identity lookup included.
899        let prepared = crate::control::ensure::prepare(&self.control, request)?;
900        let services = match &self.build_services {
901            Some(services) => Arc::clone(services),
902            None => {
903                let built: Arc<dyn crate::control::BuildServices> = Arc::new(
904                    crate::control::SignedBuildServices::new(self.control.region().clone()).await?,
905                );
906                self.build_services = Some(Arc::clone(&built));
907                built
908            }
909        };
910        let account = match &self.account {
911            Some(account) => account.clone(),
912            None => {
913                let account = services.caller_account().await?;
914                self.account = Some(account.clone());
915                account
916            }
917        };
918        let ensured =
919            crate::control::ensure::ensure(&self.control, services.as_ref(), &account, prepared)
920                .await?;
921        self.image_exists = true;
922        self.image = Some(ensured.image.clone());
923        Ok(ensured)
924    }
925
926    /// The artifact bytes to upload to the request's `code_artifact_uri`.
927    ///
928    /// The upload is the caller's on this path; [`Sandbox::ensure_image`] uploads for
929    /// itself.
930    pub fn build_artifact_for(&self, request: &CreateImageRequest) -> Result<Vec<u8>, Error> {
931        self.control.build_artifact_for(request)
932    }
933
934    /// Every local guard [`Self::build_image`] runs, callable **before** the artifact upload.
935    ///
936    /// A delegation to [`ControlPlane::preflight`], on this type so the CLI's build path
937    /// reaches it through the one sandbox it already holds. Local; zero calls. The point is
938    /// ordering: `build_image`'s guards run after the caller's upload, so a request this
939    /// client itself refuses would still cost the S3 PUT unless the caller checks first.
940    pub fn preflight(&self, request: &CreateImageRequest) -> Result<(), Error> {
941        self.control.preflight(request)
942    }
943
944    /// The ARN for `identifier`: an ARN passes through with zero calls, a bare name is
945    /// resolved through the image listing by exact match.
946    ///
947    /// A read-only delegation to [`ControlPlane::resolve_image_arn`], on this type so the
948    /// CLI's launch path reaches it through the one sandbox it already holds. It touches
949    /// none of the five state-machine variables — resolution is a question about the
950    /// account, not about this VM's lifecycle.
951    pub async fn resolve_image_arn(&self, identifier: &str) -> Result<String, Error> {
952        self.control.resolve_image_arn(identifier).await
953    }
954
955    /// The image with exactly `name`, or `None`. Read-only; see
956    /// [`ControlPlane::find_image_by_name`] for the pagination and exact-match rules.
957    pub async fn find_image_by_name(
958        &self,
959        name: &str,
960    ) -> Result<Option<crate::control::ops::MicrovmImageSummaryWire>, Error> {
961        self.control.find_image_by_name(name).await
962    }
963
964    /// The content hash `build --reuse` keys an image name to. Local; zero calls.
965    pub fn artifact_content_hash_for(&self, request: &CreateImageRequest) -> String {
966        self.control.artifact_content_hash_for(request)
967    }
968
969    /// The versions of a managed base image, for pinning
970    /// [`CreateImageRequest::base_image_version`].
971    ///
972    /// A read-only delegation to [`crate::control::ControlPlane::managed_base_versions`], on
973    /// this type for the reason [`Self::resolve_image_arn`] gives: it touches none of the five
974    /// state-machine variables, because a question about AWS's published bases is not a
975    /// question about this VM's lifecycle.
976    pub async fn managed_base_versions(
977        &self,
978        base_image_arn: &str,
979    ) -> Result<Vec<crate::control::ops::ManagedMicrovmImageVersionWire>, Error> {
980        self.control.managed_base_versions(base_image_arn).await
981    }
982
983    /// One version's configuration, state, and availability status. Read-only.
984    pub async fn image_version(
985        &self,
986        identifier: &str,
987        version: &str,
988    ) -> Result<crate::control::ops::MicrovmImageVersionSummaryWire, Error> {
989        self.control.get_image_version(identifier, version).await
990    }
991
992    // ── run (STATE-1, STATE-2, STATE-3) ──────────────────────────────────────
993
994    /// Launches a MicroVM, waits for RUNNING, and returns its session.
995    ///
996    /// # The three state requirements this is
997    ///
998    /// STATE-1: the accepted launch moves the lifecycle to PENDING and records the image as
999    /// existing. STATE-2: the platform reporting RUNNING is what marks the token installed
1000    /// — not the launch call, because the run hook is what delivers it and a launch that
1001    /// dies during startup delivered nothing. STATE-3: `bootstrap_count` is incremented
1002    /// exactly here, and a second `run` on the same sandbox is refused, which is what makes
1003    /// "at most once per VM lifetime" a property of the type rather than of a caller's
1004    /// discipline.
1005    ///
1006    /// The agent token rides in `runHookPayload`, which is what keeps it out of the shared
1007    /// image snapshot. That is safe because the platform forwards no external traffic until
1008    /// the run hook returns 200, so a per-VM secret delivered at launch wins the
1009    /// first-writer race through the endpoint.
1010    pub async fn run(&mut self, request: RunRequest) -> Result<&mut Session, Error> {
1011        self.refuse_detached("run")?;
1012        // STATE-3's local half. A sandbox that has already bootstrapped cannot bootstrap
1013        // again, and the refusal is here rather than in a comment because `run` twice is
1014        // the plausible mistake — a retry loop around a launch that timed out.
1015        if self.bootstrap_count > 0 || self.microvm.is_some() {
1016            return Err(Error::invalid_arg(format!(
1017                "this sandbox has already launched a VM ({} bootstrap(s), lifecycle {}), and the \
1018                 agent token is installed at most once per VM lifetime (STATE-3). A second VM \
1019                 needs a second Sandbox — reusing this one would either re-deliver a run-hook \
1020                 payload to a daemon whose one-shot bootstrap refuses it, or silently address \
1021                 two guests through one handle.",
1022                self.bootstrap_count, self.lifecycle,
1023            )));
1024        }
1025
1026        // The network options, refused and classified by the one function a harness asks
1027        // before a launch (BIND-13), so its answer and this launch cannot disagree. Opposite
1028        // intents are refused rather than resolved by a precedence rule nobody would find:
1029        // `--egress` asks the platform for outbound network and `--deny-egress` asks the
1030        // guest's own clients to refuse it, so a launch carrying both would report `open`
1031        // while its workload's tools failed closed.
1032        let posture = crate::control::egress_posture_for(
1033            request.egress,
1034            &request.egress_network_connectors,
1035            request.deny_egress,
1036            Some(self.control.region()),
1037        )?;
1038
1039        let Some(identifier) = request
1040            .image_identifier
1041            .clone()
1042            .or_else(|| self.image.as_ref().map(|image| image.identifier.clone()))
1043        else {
1044            return Err(Error::new(
1045                ErrorKind::Precondition,
1046                "no image to launch: pass RunRequest::with_image or call build_image first."
1047                    .to_string(),
1048            ));
1049        };
1050
1051        if request.client_token.is_some() && (request.agent_token.is_none() || request.identity) {
1052            return Err(Error::invalid_arg(
1053                "a stable client_token requires an explicit agent_token and identity=false; persist identical launch parameters before retrying",
1054            ));
1055        }
1056        let agent_token = request.agent_token.clone().unwrap_or_else(mint_agent_token);
1057        // Generated before the payload because the payload carries its two public-facing
1058        // fields. The VM's secret half lives exactly as long as this binding: `keep()` below
1059        // drops it before the launch call returns.
1060        let launch_identity = if request.identity {
1061            Some(crate::identity::LaunchIdentity::generate()?)
1062        } else {
1063            None
1064        };
1065        // Checked even though this builds the JSON itself, because neither half is ours:
1066        // the token may be caller-supplied, so someone passing a signed blob rather than a
1067        // bearer token is exactly who this catches (TRAP-5), and the launch env is entirely
1068        // the caller's. This is the pre-flight refusal — over-ceiling fails here with the
1069        // byte count, before the launch, rather than as a `ValidationException` on a member
1070        // the caller did not know they were filling.
1071        let payload = RunHookPayload::for_launch_with_identity(
1072            &agent_token,
1073            &request.effective_launch_env(),
1074            launch_identity.as_ref(),
1075        )?;
1076
1077        let mut wire = RunMicrovmRequest::new(&identifier, payload);
1078        wire.image_version = request.image_version.clone();
1079        wire.execution_role_arn = request.execution_role_arn.clone();
1080        wire.egress_network_connectors = request.egress_network_connectors.clone();
1081        wire.max_idle_sec = request.max_idle_sec;
1082        wire.suspended_sec = request.suspended_sec;
1083        wire.auto_resume = request.auto_resume;
1084        wire.max_duration_sec = request.max_duration_sec;
1085        wire.token_scope = request.token_scope.clone();
1086        wire.client_token = request.client_token.clone();
1087        wire.logging = request.logging.clone();
1088        if request.egress {
1089            wire = wire.with_egress();
1090        }
1091        if request.shell {
1092            wire = wire.with_shell();
1093        }
1094
1095        let launched = self.control.run_microvm(wire).await?;
1096
1097        // The durable half of the identity, kept the moment the launch is accepted: the host
1098        // secret and the VM's public pin. The VM's own secret ends here — `keep` drops it —
1099        // so from this line on the host can verify its VM and impersonate it never.
1100        self.tunnel_identity = launch_identity.map(crate::identity::LaunchIdentity::keep);
1101
1102        // STATE-1: the launch was accepted.
1103        self.set_lifecycle(Lifecycle::Pending);
1104        self.image_exists = true;
1105        // Recorded here rather than after the wait, because the window the idlePolicy
1106        // enforces was set by *this* request and a launch that then fails still leaves a VM
1107        // the caller may have to reason about.
1108        self.suspended_window = Some(Duration::from_secs(u64::from(request.suspended_sec)));
1109        self.idle_window = Some(Duration::from_secs(u64::from(request.max_idle_sec)));
1110        self.microvm = Some(launched);
1111        // A caller-supplied client token can adopt the VM an earlier attempt launched, which
1112        // may have idle-suspended since (#195); a minted token always launches afresh.
1113        self.launch_adoptable = request.client_token.is_some();
1114        self.launch_posture = Some(posture);
1115        self.pending_agent_token = Some(agent_token);
1116
1117        if request.wait {
1118            return self.wait_until_running(request.ready_timeout).await;
1119        }
1120        // Not waiting: the endpoint is in the launch reply, so the session is addressable
1121        // now, and a daemon request before RUNNING fails like any other unready request.
1122        self.build_session()?;
1123        Ok(self.session.as_mut().expect("just assigned"))
1124    }
1125
1126    /// The session for the launched VM, from its endpoint and the launch's agent token.
1127    fn build_session(&mut self) -> Result<(), Error> {
1128        let (Some(vm), Some(agent_token)) = (&self.microvm, &self.pending_agent_token) else {
1129            return Err(Error::new(
1130                ErrorKind::Precondition,
1131                "no launch to build a session for",
1132            ));
1133        };
1134        let minter = Arc::new(ControlPlaneMinter {
1135            control: Arc::clone(&self.control),
1136            microvm_id: vm.id.clone(),
1137        });
1138        let mut builder = Session::builder(vm.endpoint.clone(), agent_token.clone())
1139            .with_minter(minter)
1140            .with_port(self.control.port())
1141            .with_egress_posture(self.egress_posture());
1142        if let Some(backend) = &self.session_backend {
1143            builder = builder.with_backend(Arc::clone(backend));
1144        }
1145        self.session = Some(builder.build()?);
1146        Ok(())
1147    }
1148
1149    /// Waits for a launch accepted by [`Sandbox::run`] to reach RUNNING.
1150    ///
1151    /// `run` calls this itself unless [`RunRequest::wait`] was off. A launch that adopted an
1152    /// existing VM through its client token resumes it if it idle-suspended (#195); a fresh
1153    /// launch that reaches any terminal state first fails fast with `stateReason` (TRAP-8).
1154    pub async fn wait_until_running(&mut self, timeout: Duration) -> Result<&mut Session, Error> {
1155        self.refuse_detached("wait until running")?;
1156        let id = self.require_microvm("wait_until_running")?;
1157        if self.lifecycle != Lifecycle::Pending {
1158            return Err(Error::invalid_arg(format!(
1159                "microvm {id} is {}; wait_until_running finishes a launch that is still PENDING",
1160                self.lifecycle,
1161            )));
1162        }
1163        let opts = WaitOpts {
1164            timeout,
1165            poll_interval: LIFECYCLE_POLL_INTERVAL,
1166            stall_grace: Duration::MAX,
1167        };
1168        let running = if self.launch_adoptable {
1169            self.control.wait_for_launch(&id, opts).await?
1170        } else {
1171            self.control.wait_for_running(&id, opts).await?
1172        };
1173
1174        // STATE-2. The platform reported the run hook succeeded, so the token is in the
1175        // guest's memory — and this is the one place that counts it (STATE-3).
1176        self.set_lifecycle(Lifecycle::Running);
1177        self.token_installed = true;
1178        self.bootstrap_count += 1;
1179        self.microvm = Some(running);
1180        if self.session.is_none() {
1181            self.build_session()?;
1182        }
1183        Ok(self.session.as_mut().expect("just assigned"))
1184    }
1185
1186    // ── adopt (STATE-3, with the lifecycle read from the service) ────────────
1187
1188    /// A sandbox for a VM another process launched, rebuilt from its private record.
1189    ///
1190    /// The durable-workflow shape: every step runs in a fresh process, and each one needs
1191    /// suspend, resume, and terminate, not just the exec and file surface
1192    /// [`Session::attach`] gives. The lifecycle is read from `GetMicrovm` rather than
1193    /// assumed, so every guard below starts from what the service reports.
1194    ///
1195    /// # Bootstrap is counted, never repeated (STATE-3)
1196    ///
1197    /// The VM was bootstrapped by the launch that created it, so an adopted sandbox refuses
1198    /// [`Sandbox::run`] and never sends a run-hook payload. A VM adopted while still PENDING
1199    /// finishes through [`Sandbox::wait_until_running`], which counts the bootstrap once
1200    /// when the service reports RUNNING, the same as a launch this sandbox made itself.
1201    ///
1202    /// # The suspended window (STATE-12)
1203    ///
1204    /// The window comes from the `idlePolicy` `GetMicrovm` reports. A VM adopted while
1205    /// already SUSPENDED has no suspend time this client observed, so the local check has
1206    /// nothing to measure and the service answers the resume instead.
1207    ///
1208    /// `endpoint` must agree with the service's when the service reports one: a mismatch
1209    /// means the identifiers came from two different records. `agent_token` is the bearer
1210    /// credential the VM was launched with; it never appears in `Debug` or an error.
1211    pub async fn adopt(
1212        control: ControlPlane,
1213        microvm_id: impl Into<String>,
1214        endpoint: impl Into<String>,
1215        agent_token: impl Into<String>,
1216    ) -> Result<Self, Error> {
1217        let microvm_id = microvm_id.into();
1218        let endpoint = endpoint.into();
1219        let agent_token = agent_token.into();
1220        if agent_token.is_empty() {
1221            return Err(Error::invalid_arg(
1222                "adopt needs the agent token the VM was launched with: its daemon accepts only \
1223                 that bearer, and adopting without it would build a handle every exec refuses.",
1224            ));
1225        }
1226        let mut vm = control.get_microvm(&microvm_id).await?;
1227        if !endpoint.is_empty() && !vm.endpoint.is_empty() && vm.endpoint != endpoint {
1228            return Err(Error::invalid_arg(format!(
1229                "microvm {microvm_id} reports endpoint {}, not {endpoint}: the id and endpoint \
1230                 came from different records, and a session built from them would address one \
1231                 VM while its lifecycle calls reached another.",
1232                vm.endpoint,
1233            )));
1234        }
1235        if vm.endpoint.is_empty() {
1236            vm.endpoint = endpoint;
1237        }
1238        let Some(lifecycle) = Lifecycle::from_service(&vm.state) else {
1239            return Err(Error::new(
1240                ErrorKind::Platform,
1241                format!(
1242                    "microvm {microvm_id} reports state {}, which this client does not know; \
1243                     refusing to guess which lifecycle guards apply.",
1244                    vm.state,
1245                ),
1246            ));
1247        };
1248
1249        let mut sandbox = Self::with_control_plane(control);
1250        sandbox.adopted = true;
1251        sandbox.set_lifecycle(lifecycle);
1252        // The window the VM was launched with, so a keepalive on the adopted session paces
1253        // itself against the real idle policy rather than the platform minimum.
1254        sandbox.idle_window = vm
1255            .idle_policy
1256            .as_ref()
1257            .map(|policy| Duration::from_secs(u64::from(policy.max_idle_duration_seconds)));
1258        // A VM exists, so the image it was launched from did (STATE-1).
1259        sandbox.image_exists = true;
1260        // PENDING may still idle-suspend before this handle sees RUNNING (#195).
1261        sandbox.launch_adoptable = true;
1262        sandbox.pending_agent_token = Some(agent_token);
1263        match lifecycle {
1264            Lifecycle::Pending => {}
1265            Lifecycle::Running | Lifecycle::Suspending | Lifecycle::Suspended => {
1266                sandbox.token_installed = true;
1267                sandbox.bootstrap_count = 1;
1268            }
1269            Lifecycle::Terminating | Lifecycle::Terminated => {
1270                sandbox.bootstrap_count = 1;
1271                sandbox.was_terminated = true;
1272            }
1273        }
1274        sandbox.microvm = Some(vm);
1275        if lifecycle.is_live() {
1276            sandbox.build_session()?;
1277        }
1278        Ok(sandbox)
1279    }
1280
1281    /// [`Sandbox::adopt`] over a plane resolved for `region`, on `port` when the image's
1282    /// daemon listens somewhere other than the default. The bindings' entry point.
1283    pub async fn adopt_in(
1284        region: Region,
1285        microvm_id: impl Into<String>,
1286        endpoint: impl Into<String>,
1287        agent_token: impl Into<String>,
1288        port: Option<u16>,
1289    ) -> Result<Self, Error> {
1290        Self::adopt(
1291            Self::plane_for(region, port).await?,
1292            microvm_id,
1293            endpoint,
1294            agent_token,
1295        )
1296        .await
1297    }
1298
1299    /// A [`crate::names::NameRecord`] for this VM, to register under `name`.
1300    ///
1301    /// The egress posture is left unknown: the sandbox does not keep its launch request, and
1302    /// a record must not claim a network it cannot vouch for.
1303    pub fn name_record(&self, name: &str) -> Result<crate::names::NameRecord, Error> {
1304        let Some(vm) = self.microvm() else {
1305            return Err(Error::new(
1306                ErrorKind::Precondition,
1307                "there is no VM to name yet: launch or adopt one first",
1308            ));
1309        };
1310        let Some(session) = self.session() else {
1311            return Err(Error::new(
1312                ErrorKind::Precondition,
1313                format!(
1314                    "microvm {} has no session to take its agent token from",
1315                    vm.id
1316                ),
1317            ));
1318        };
1319        crate::names::NameRecord::new(
1320            name,
1321            vm.id.as_str(),
1322            vm.endpoint.as_str(),
1323            session.agent_token(),
1324            self.control.region().as_str(),
1325        )
1326    }
1327
1328    /// Adopts the VM registered as `name` in `store`: [`Sandbox::adopt`] from a record.
1329    ///
1330    /// `region`, when given, must match the record's; see [`crate::names::resolve`].
1331    pub async fn from_name(
1332        store: &dyn crate::names::NameStore,
1333        name: &str,
1334        region: Option<Region>,
1335        port: Option<u16>,
1336    ) -> Result<Self, Error> {
1337        let record = crate::names::resolve(store, name, region.as_ref())?;
1338        let control = Self::plane_for(record.region(), port).await?;
1339        Self::adopt_record(control, record).await
1340    }
1341
1342    /// [`Sandbox::adopt`] from a [`crate::names::NameRecord`] kept in any store.
1343    pub async fn adopt_record(
1344        control: ControlPlane,
1345        record: crate::names::NameRecord,
1346    ) -> Result<Self, Error> {
1347        Self::adopt(
1348            control,
1349            record.microvm_id,
1350            record.endpoint,
1351            record.agent_token,
1352        )
1353        .await
1354    }
1355
1356    /// A control plane for `region`, on `port` when one is given.
1357    pub(crate) async fn plane_for(
1358        region: Region,
1359        port: Option<u16>,
1360    ) -> Result<ControlPlane, Error> {
1361        let control = ControlPlane::new(region).await?;
1362        match port {
1363            Some(port) => control.with_port(port),
1364            None => Ok(control),
1365        }
1366    }
1367
1368    /// Hands the VM off to another process and returns what that process needs to adopt it.
1369    ///
1370    /// For a workflow whose steps run in different processes (a durable function's launch
1371    /// step, then later steps that [`Sandbox::adopt`]): the launching process calls this
1372    /// instead of dropping the sandbox, which would warn that a live VM was abandoned. The
1373    /// VM keeps running and nothing is sent to AWS. From here on this sandbox is inert — its
1374    /// session is dropped and `run`, `wait_until_running`, `suspend`, `resume`, and
1375    /// `terminate` are refused — because the adopter now owns the lifecycle, and a second
1376    /// handle driving it would bypass the adopter's guards.
1377    ///
1378    /// Refused (`Precondition`) when there is no live VM to hand off: nothing launched yet,
1379    /// already torn down or terminated, or already detached.
1380    pub fn detach(&mut self) -> Result<Detached, Error> {
1381        self.refuse_detached("detach")?;
1382        let Some(vm) = self.microvm.as_ref() else {
1383            return Err(Error::new(
1384                ErrorKind::Precondition,
1385                "nothing to detach: this sandbox has not launched a VM.",
1386            ));
1387        };
1388        if self.torn_down || !self.lifecycle.is_live() {
1389            return Err(Error::new(
1390                ErrorKind::Precondition,
1391                format!(
1392                    "microvm {} is {} and there is no live VM to hand off; only a PENDING, \
1393                     RUNNING, or SUSPENDED VM can be adopted.",
1394                    vm.id, self.lifecycle,
1395                ),
1396            ));
1397        }
1398        let agent_token = self
1399            .session
1400            .as_ref()
1401            .map(|session| session.agent_token().to_string())
1402            .or_else(|| self.pending_agent_token.clone())
1403            .ok_or_else(|| {
1404                Error::new(
1405                    ErrorKind::Precondition,
1406                    format!(
1407                        "microvm {} has no agent token to hand off; an adopter could not \
1408                         reach its daemon.",
1409                        vm.id
1410                    ),
1411                )
1412            })?;
1413        let detached = Detached {
1414            microvm_id: vm.id.clone(),
1415            endpoint: vm.endpoint.clone(),
1416            region: self.control.region().clone(),
1417            port: self.control.port(),
1418            agent_token,
1419        };
1420        self.detached = true;
1421        self.session = None;
1422        self.pending_agent_token = None;
1423        Ok(detached)
1424    }
1425
1426    /// Whether [`Sandbox::detach`] handed this sandbox's VM to another process.
1427    pub fn detached(&self) -> bool {
1428        self.detached
1429    }
1430
1431    /// Refuses `what` on a sandbox whose VM was handed off.
1432    fn refuse_detached(&self, what: &str) -> Result<(), Error> {
1433        if !self.detached {
1434            return Ok(());
1435        }
1436        let id = self.microvm.as_ref().map_or("?", |vm| vm.id.as_str());
1437        Err(Error::new(
1438            ErrorKind::Precondition,
1439            format!(
1440                "cannot {what}: microvm {id} was detached and belongs to whoever adopts it. \
1441                 Adopt it with `Sandbox::adopt` and {what} through that handle."
1442            ),
1443        ))
1444    }
1445
1446    /// Whether this sandbox was built by [`Sandbox::adopt`] rather than by its own launch.
1447    pub fn adopted(&self) -> bool {
1448        self.adopted
1449    }
1450
1451    // ── suspend (STATE-4, STATE-5, STATE-6) ──────────────────────────────────
1452
1453    /// Freezes the VM and waits for the platform to report it.
1454    ///
1455    /// A freeze and restore rather than a stop and start: the guest keeps its memory, so the
1456    /// token, the filesystem, and every exec record survive. The one thing that does not is
1457    /// the guest's view of time — it observes the whole suspension as a single jump, so any
1458    /// timeout, lease, or TLS session a running command holds expires at once on resume.
1459    ///
1460    /// # STATE-5 is checked before the wire, not after
1461    ///
1462    /// A suspend from anything but RUNNING is refused here with **zero** control-plane
1463    /// calls. That is the observable difference between this and a client that lets AWS
1464    /// answer, and it is what the test asserts on.
1465    pub async fn suspend(&mut self) -> Result<(), Error> {
1466        self.refuse_detached("suspend")?;
1467        let id = self.require_microvm("suspend")?;
1468
1469        // STATE-5.
1470        if self.lifecycle != Lifecycle::Running {
1471            return Err(Error::invalid_arg(format!(
1472                "microvm {id} is {} and a suspend is only issued from RUNNING (STATE-5). Refused \
1473                 here rather than by the service, because the service's answer about a \
1474                 non-running id does not say which of the two things went wrong — and a suspend \
1475                 issued from SUSPENDED is a caller who believes they resumed.",
1476                self.lifecycle,
1477            )));
1478        }
1479
1480        // STATE-4: accepted, so the lifecycle moves before the wait. Acceptance is the
1481        // wire call succeeding, so the assignment comes after it: moving to SUSPENDING
1482        // first would leave a failed call (a throttle, a dead transport) stuck in a state
1483        // neither suspend nor resume accepts, bricking the handle over one bad request.
1484        self.control.suspend(&id).await?;
1485        self.set_lifecycle(Lifecycle::Suspending);
1486        // Stamped after the call and before the wait, not after the wait: the idlePolicy's
1487        // window starts when the platform begins suspending, so timing it from SUSPENDED
1488        // would under-count the transition and call a closed window open.
1489        self.suspended_at = Some(self.control.clock().elapsed());
1490
1491        let settled = self
1492            .control
1493            .wait_for_state(
1494                &id,
1495                &crate::control::microvm::SUSPEND_WANTED,
1496                &[],
1497                self.lifecycle_wait(),
1498            )
1499            .await?;
1500
1501        // STATE-6, and the TERMINATED case beside it. A VM that dies while suspending is a
1502        // state to report rather than an exception out of the middle of a teardown, so the
1503        // wait *wants* TERMINATED — and recording it here is what stops a resume from being
1504        // offered afterwards (STATE-11).
1505        let settled_lifecycle = match settled.state.as_str() {
1506            "SUSPENDED" => Lifecycle::Suspended,
1507            "TERMINATED" => {
1508                self.was_terminated = true;
1509                Lifecycle::Terminated
1510            }
1511            other => {
1512                return Err(Error::new(
1513                    ErrorKind::Platform,
1514                    format!(
1515                        "the suspend wait returned {other}, which is neither SUSPENDED nor \
1516                         TERMINATED — and those two are what this client asked for."
1517                    ),
1518                ));
1519            }
1520        };
1521        self.set_lifecycle(settled_lifecycle);
1522        self.microvm = Some(settled);
1523        Ok(())
1524    }
1525
1526    // ── resume (STATE-7, STATE-8, STATE-12) ──────────────────────────────────
1527
1528    /// Thaws the VM and returns a usable session.
1529    ///
1530    /// # What is deliberately not re-delivered (STATE-7)
1531    ///
1532    /// Nothing. No run-hook payload, no token, no bootstrap. The in-memory token survived
1533    /// the freeze, and re-delivering it would hit the daemon's one-shot bootstrap and be
1534    /// refused — a 409 that reads like a broken VM.
1535    ///
1536    /// # The window is checked first (STATE-12)
1537    ///
1538    /// Before any wire call, because the answer is already known and calling costs the poll
1539    /// timeout to learn something worse. The falsification is a service reporting
1540    /// TERMINATED: without this check the resume burns the full deadline and then reports a
1541    /// state the client could have named at once.
1542    ///
1543    /// # The proxy token is dropped (STATE-8)
1544    ///
1545    /// [`Session::rebind`] invalidates it. The endpoint URL does not change across
1546    /// suspend/resume, so the rebind is usually a no-op on the URL — but a token minted
1547    /// against the pre-suspend instance may no longer validate, and that rejection reads
1548    /// exactly like a dead daemon.
1549    pub async fn resume(&mut self) -> Result<&mut Session, Error> {
1550        self.refuse_detached("resume")?;
1551        let id = self.require_microvm("resume")?;
1552
1553        // STATE-11's local half: a terminated VM never returns to RUNNING, so the refusal
1554        // comes before the window check and before any call.
1555        if self.was_terminated || self.lifecycle == Lifecycle::Terminated {
1556            return Err(Error::invalid_arg(format!(
1557                "microvm {id} was terminated, and a terminated VM never returns to RUNNING \
1558                 (STATE-11). There is nothing to resume: the guest's memory is gone, so even a \
1559                 call the service accepted would hand back a different machine."
1560            )));
1561        }
1562        if self.lifecycle != Lifecycle::Suspended {
1563            return Err(Error::invalid_arg(format!(
1564                "microvm {id} is {} and a resume is only issued from SUSPENDED (STATE-7).",
1565                self.lifecycle,
1566            )));
1567        }
1568
1569        // STATE-12, first and locally.
1570        self.require_open_suspended_window(&id)?;
1571
1572        self.control.resume(&id).await?;
1573        // `fail_on` is the *dead* states rather than the terminal ones: SUSPENDED is the
1574        // state this call was made from, so failing on it would fail every resume. A VM the
1575        // idlePolicy terminated during suspension never reaches RUNNING, and waiting only
1576        // for RUNNING there burns the full timeout and then reports a timeout message
1577        // hiding a cause the service had already stated in `stateReason`.
1578        let running = self
1579            .control
1580            .wait_for_state(
1581                &id,
1582                &["RUNNING"],
1583                &crate::constants::DEAD_STATES,
1584                self.lifecycle_wait(),
1585            )
1586            .await?;
1587
1588        self.set_lifecycle(Lifecycle::Running);
1589        // STATE-8, through the endpoint the service just reported rather than the one held:
1590        // the URL is measured not to change, and reading it from the response is what makes
1591        // that a fact this code depends on rather than an assumption it encodes.
1592        let endpoint = running.endpoint.clone();
1593        self.microvm = Some(running);
1594        if let Some(session) = self.session.as_mut() {
1595            session.rebind(endpoint);
1596        }
1597        // Cleared on success so the next cycle's window is measured from the next suspend.
1598        // Leaving it set would accumulate every suspension's elapsed time into one total and
1599        // reject a resume whose own window is wide open.
1600        self.suspended_at = None;
1601
1602        self.session.as_mut().ok_or_else(|| {
1603            Error::new(
1604                ErrorKind::Unexpected,
1605                "the VM resumed but this sandbox holds no session, which run() always builds"
1606                    .to_string(),
1607            )
1608        })
1609    }
1610
1611    /// Rejects a resume the launch-time `idlePolicy` has already made impossible (STATE-12).
1612    ///
1613    /// The message names the elapsed time, the window, and the `idlePolicy` finding, because
1614    /// "cannot resume" alone sends a reader looking for the flag that reopens it.
1615    fn require_open_suspended_window(&self, id: &str) -> Result<(), Error> {
1616        let reported = self
1617            .microvm
1618            .as_ref()
1619            .and_then(|vm| vm.idle_policy.as_ref())
1620            .map(|policy| Duration::from_secs(u64::from(policy.suspended_duration_seconds)));
1621        let (Some(window), Some(since)) = (self.suspended_window.or(reported), self.suspended_at)
1622        else {
1623            // No window or no suspend time means this sandbox cannot know how long the VM
1624            // has been suspended, and guessing would refuse a resume the service would honour.
1625            return Ok(());
1626        };
1627        let elapsed = self.control.clock().elapsed().saturating_sub(since);
1628        if elapsed <= window {
1629            return Ok(());
1630        }
1631        Err(Error::new(
1632            ErrorKind::WindowClosed,
1633            format!(
1634                "microvm {id} has been suspended {}s, past the {}s suspendedDurationSeconds \
1635                 window set at launch — the idlePolicy terminates a suspended VM once that window \
1636                 passes, so there is nothing left to resume (docs/PLATFORM.md, '`idlePolicy`'). \
1637                 Refused before ResumeMicrovm, because calling would cost the full poll timeout \
1638                 to learn the same thing less clearly. A longer window has to be set at launch \
1639                 on the next VM; there is no call that extends this one.",
1640                elapsed.as_secs(),
1641                window.as_secs(),
1642            ),
1643        ))
1644    }
1645
1646    // ── terminate (STATE-9, STATE-10) ────────────────────────────────────────
1647
1648    /// Tears down, best-effort, never erroring.
1649    ///
1650    /// Order: VM, then image, then the log group **last**. The log group is last because the
1651    /// service can recreate a group deleted before its image — and see
1652    /// [`TeardownReport::undeleted`] for why this crate names it rather than deleting it.
1653    pub async fn terminate(&mut self, opts: TeardownOpts) -> TeardownReport {
1654        let mut report = TeardownReport::default();
1655        if let Err(error) = self.refuse_detached("terminate") {
1656            // Never erroring is `terminate`'s contract, so the refusal is reported rather
1657            // than raised; the VM is untouched and still belongs to its adopter.
1658            report.failures.push(error.to_string());
1659            report.lifecycle = Some(self.lifecycle);
1660            return report;
1661        }
1662        self.torn_down = true;
1663
1664        // The session first: it holds a cached proxy token whose only remaining use would be
1665        // a request against a VM that is going away.
1666        self.session = None;
1667
1668        // 1. The VM.
1669        if let Some(id) = self.microvm.as_ref().map(|vm| vm.id.clone()) {
1670            // STATE-9. Recorded before the call, so a terminate whose call fails still marks
1671            // the VM as one this client asked to destroy — which is what stops a later
1672            // resume (STATE-11) rather than leaving the sandbox looking resumable.
1673            self.set_lifecycle(Lifecycle::Terminating);
1674            self.was_terminated = true;
1675
1676            match self.control.terminate(&id).await {
1677                Ok(()) => report.terminate_accepted = true,
1678                Err(error) => {
1679                    report.failures.push(format!("terminate {id}: {error}"));
1680                    report.undeleted.push(id.clone());
1681                }
1682            }
1683
1684            if report.terminate_accepted
1685                && let Some(timeout) = opts.wait_for_terminated
1686            {
1687                let wait = WaitOpts {
1688                    timeout,
1689                    poll_interval: LIFECYCLE_POLL_INTERVAL,
1690                    stall_grace: Duration::MAX,
1691                };
1692                match self
1693                    .control
1694                    .wait_for_state(&id, &["TERMINATED"], &[], wait)
1695                    .await
1696                {
1697                    // STATE-10.
1698                    Ok(settled) => {
1699                        self.set_lifecycle(Lifecycle::Terminated);
1700                        self.microvm = Some(settled);
1701                    }
1702                    // Not a leak: the platform accepted the terminate, so the VM is on its
1703                    // way out and the lifecycle stays TERMINATING honestly.
1704                    Err(error) => report
1705                        .failures
1706                        .push(format!("waiting for {id} to reach TERMINATED: {error}")),
1707                }
1708            }
1709        }
1710
1711        // 2. The image, retrying — an image in CREATING refuses deletion and a VM still
1712        //    terminating holds a reference.
1713        if opts.delete_image {
1714            match self.image.as_ref().map(|image| image.identifier.clone()) {
1715                Some(identifier) => {
1716                    let deleted = self
1717                        .control
1718                        .delete_image(&identifier, opts.delete_attempts, opts.delete_backoff)
1719                        .await;
1720                    report.image_deleted = Some(deleted);
1721                    if deleted {
1722                        self.image_exists = false;
1723                    } else {
1724                        report.failures.push(format!(
1725                            "the image {identifier} survived {} delete attempts",
1726                            opts.delete_attempts.max(1)
1727                        ));
1728                        report.undeleted.push(identifier);
1729                    }
1730                }
1731                None => report.image_deleted = Some(false),
1732            }
1733        }
1734
1735        // 3. The log group, LAST. Named rather than deleted; see TeardownReport::undeleted.
1736        if opts.delete_log_group
1737            && let Some(group) = self.image.as_ref().map(Image::build_log_group)
1738        {
1739            report.failures.push(format!(
1740                "the build log group {group} was not deleted: CloudWatch Logs is not in this \
1741                 crate's dependency set, so it is reported rather than removed. It is \
1742                 service-created, which means no Terraform stack owns it and `terraform destroy` \
1743                 leaves it behind — six accumulated before anyone noticed."
1744            ));
1745            report.undeleted.push(group);
1746        }
1747
1748        report.lifecycle = Some(self.lifecycle);
1749        report
1750    }
1751
1752    // ── internals ────────────────────────────────────────────────────────────
1753
1754    /// The VM id, or a precondition error naming what was attempted.
1755    fn require_microvm(&self, what: &str) -> Result<String, Error> {
1756        self.microvm
1757            .as_ref()
1758            .map(|vm| vm.id.clone())
1759            .ok_or_else(|| {
1760                Error::new(
1761                    ErrorKind::Precondition,
1762                    format!("nothing to {what}: this sandbox has not launched a VM."),
1763                )
1764            })
1765    }
1766
1767    /// Five minutes at five-second polls, for suspend, resume, and terminate.
1768    fn lifecycle_wait(&self) -> WaitOpts {
1769        WaitOpts {
1770            timeout: DEFAULT_LIFECYCLE_TIMEOUT,
1771            poll_interval: LIFECYCLE_POLL_INTERVAL,
1772            stall_grace: Duration::MAX,
1773        }
1774    }
1775}
1776
1777/// Warns about a live VM rather than tearing it down.
1778///
1779/// See the module docs: `Drop` cannot await, so a teardown here would either deadlock
1780/// inside a runtime or race the process exit. The warning names the id, because the only
1781/// useful thing a drop can do is tell whoever reads stderr what to go delete.
1782///
1783/// A plain write to stderr rather than a log macro, because this crate has no logging
1784/// dependency, and taking one on to warn about a leak would be a dependency for a diagnostic.
1785/// Not `eprintln!`: it panics when stderr's reader has gone, and a panic in `drop` during an
1786/// unwind aborts the process (CLI-7). The error is ignored; there is nowhere left to report it.
1787impl Sandbox {
1788    /// The warning `Drop` writes for a live VM nobody took responsibility for, or `None`
1789    /// when dropping is not a leak: torn down, adopted (its launcher owns it), or detached
1790    /// (its adopter owns it).
1791    fn drop_warning(&self) -> Option<String> {
1792        if self.torn_down || self.adopted || self.detached {
1793            return None;
1794        }
1795        let vm = self.microvm.as_ref()?;
1796        self.lifecycle.is_live().then(|| {
1797            format!(
1798                "warning: the Sandbox for microvm {} was dropped in {} without terminate() or \
1799                 detach(). Nothing was torn down — Drop cannot await, so a teardown here would \
1800                 deadlock inside a runtime. The VM bills until its maximumDurationInSeconds \
1801                 ceiling: terminate it with `microvm terminate {}`.",
1802                vm.id, self.lifecycle, vm.id,
1803            )
1804        })
1805    }
1806}
1807
1808impl Drop for Sandbox {
1809    fn drop(&mut self) {
1810        use std::io::Write as _;
1811        if let Some(warning) = self.drop_warning() {
1812            let _ = writeln!(std::io::stderr(), "{warning}");
1813        }
1814    }
1815}
1816
1817/// A fresh per-VM bearer token: 32 bytes of randomness as 64 hex characters.
1818///
1819/// `getrandom` reaches the kernel CSPRNG directly. The hand-rolled `/dev/urandom`
1820/// read this replaces carried a clock-mixing fallback for a failed read; getrandom
1821/// fails only when the OS pool is genuinely unavailable, and minting a bearer token
1822/// from a clock in that state would be worse than refusing the launch.
1823fn mint_agent_token() -> String {
1824    let mut bytes = [0u8; 32];
1825    getrandom::fill(&mut bytes).expect("the OS random pool is available");
1826    const_hex::encode(bytes)
1827}
1828
1829#[cfg(test)]
1830mod tests {
1831    use super::*;
1832    use crate::control::fake::{self as fake, Answer, FakeControlPlane, TestClock};
1833    use crate::control::transport::Method;
1834
1835    /// A sandbox over the contract recorder, plus the two handles a test asserts through.
1836    ///
1837    /// The recorder is T-W2-4's, deliberately: its answers are literal JSON in the service
1838    /// model's spelling rather than values serialized from this crate's types, so a member
1839    /// this lane misreads cannot be misread identically by the fake.
1840    fn planted() -> (Sandbox, Arc<FakeControlPlane>, Arc<TestClock>) {
1841        let recorder = Arc::new(FakeControlPlane::new());
1842        let clock = Arc::new(TestClock::new());
1843        let plane = ControlPlane::with_transport(
1844            Arc::clone(&recorder) as Arc<dyn crate::control::transport::Transport>,
1845            Region::UsEast1,
1846            Arc::clone(&clock) as Arc<dyn crate::control::Clock>,
1847        );
1848        (Sandbox::with_control_plane(plane), recorder, clock)
1849    }
1850
1851    /// Queues everything a launch to RUNNING needs.
1852    fn answer_launch(recorder: &FakeControlPlane) {
1853        recorder
1854            .answer(
1855                "RunMicrovm",
1856                Answer::ok(fake::microvm_response("PENDING", None)),
1857            )
1858            .answer(
1859                "GetMicrovm",
1860                Answer::ok(fake::microvm_response("RUNNING", None)),
1861            )
1862            .answer(
1863                "CreateMicrovmAuthToken",
1864                Answer::ok(fake::auth_token_response("proxy-token")),
1865            );
1866    }
1867
1868    #[tokio::test]
1869    async fn vpc_egress_configuration_survives_the_sandbox_launch() {
1870        let (mut sandbox, recorder, _) = planted();
1871        answer_launch(&recorder);
1872        let arn = "arn:aws:lambda:us-east-1:123456789012:network-connector:isolated-vpc";
1873        let request = RunRequest::new()
1874            .with_image("arn:image")
1875            .with_egress_network_connector(arn);
1876        assert_eq!(
1877            request.egress_posture(),
1878            crate::control::EgressPosture::Unsealed
1879        );
1880        sandbox.run(request).await.expect("launch");
1881        let body = recorder.first_body("RunMicrovm");
1882        assert_eq!(body["egressNetworkConnectors"], serde_json::json!([arn]));
1883    }
1884
1885    /// **BIND-12: the launched session carries its request's posture**, the value
1886    /// `egress_posture_for` answers for the same options and the CLI envelope reports
1887    /// (`guards.rs` holds that half). Each of the four launchable shapes.
1888    ///
1889    /// **Falsification** — 2026-09-24. Build the session without `with_egress_posture` and
1890    /// the `open` and `best-effort` rows read `unsealed`; restored.
1891    #[tokio::test]
1892    async fn a_launched_session_reports_its_requests_posture() {
1893        use crate::control::EgressPosture;
1894        let arn = "arn:aws:lambda:us-east-1:123456789012:network-connector:isolated-vpc";
1895        let rows = [
1896            (RunRequest::new(), EgressPosture::Unsealed),
1897            (RunRequest::new().with_egress(), EgressPosture::Open),
1898            (
1899                RunRequest::new().with_deny_egress(),
1900                EgressPosture::BestEffort,
1901            ),
1902            (
1903                RunRequest::new().with_egress_network_connector(arn),
1904                EgressPosture::Unsealed,
1905            ),
1906        ];
1907        for (request, expected) in rows {
1908            let answered = crate::control::egress_posture_for(
1909                request.egress,
1910                &request.egress_network_connectors,
1911                request.deny_egress,
1912                Some(&Region::UsEast1),
1913            )
1914            .expect("a launchable request");
1915            assert_eq!(answered, expected);
1916            let (mut sandbox, recorder, _) = planted();
1917            answer_launch(&recorder);
1918            let session = sandbox
1919                .run(request.with_image("arn:image"))
1920                .await
1921                .expect("the launch reaches RUNNING");
1922            assert_eq!(session.egress_posture(), expected, "BIND-12");
1923            assert_eq!(sandbox.egress_posture(), expected);
1924            sandbox.detach().expect("hand the scripted VM off quietly");
1925        }
1926    }
1927
1928    /// **BIND-12: a session that does not hold its launch options reports `unsealed`**, the
1929    /// weakest true claim, whatever the VM was launched with: an adopted VM and a session
1930    /// attached directly both lack the request.
1931    #[tokio::test]
1932    async fn a_session_without_its_launch_options_reports_unsealed() {
1933        use crate::control::EgressPosture;
1934        let (sandbox, _recorder, _clock) = adopted_in("RUNNING").await;
1935        assert_eq!(
1936            sandbox.session().expect("RUNNING").egress_posture(),
1937            EgressPosture::Unsealed
1938        );
1939        assert_eq!(sandbox.egress_posture(), EgressPosture::Unsealed);
1940        let direct = Session::direct(ADOPT_ENDPOINT, ADOPT_TOKEN).expect("a direct session");
1941        assert_eq!(direct.egress_posture(), EgressPosture::Unsealed);
1942    }
1943
1944    /// A launched sandbox in RUNNING, which is where four of the twelve keys start.
1945    async fn launched() -> (Sandbox, Arc<FakeControlPlane>, Arc<TestClock>) {
1946        let (mut sandbox, recorder, clock) = planted();
1947        answer_launch(&recorder);
1948        sandbox
1949            .run(RunRequest::new().with_image("arn:image"))
1950            .await
1951            .expect("the launch reaches RUNNING");
1952        (sandbox, recorder, clock)
1953    }
1954
1955    /// **Detach hands off, and nothing warns or moves afterwards.** A launched sandbox
1956    /// warns on drop; once detached it returns what an adopter needs (with the token kept
1957    /// out of `Debug`), stops warning, makes no further control-plane call, and refuses every
1958    /// transition — including `terminate`, which reports the refusal instead of raising.
1959    ///
1960    /// **Falsification** — 2026-09-24. Leave `detached` out of `drop_warning`'s early
1961    /// return and the "no warning after detach" assertion goes red; drop the
1962    /// `refuse_detached` guard from `suspend` and the zero-calls assertion goes red (the
1963    /// suspend reaches `SuspendMicrovm`). Both restored.
1964    #[tokio::test]
1965    async fn a_detached_sandbox_hands_off_its_vm_and_stays_quiet() {
1966        let (mut sandbox, recorder, _clock) = launched().await;
1967        assert!(
1968            sandbox.drop_warning().is_some(),
1969            "a launched, live VM warns on drop"
1970        );
1971        let token = sandbox
1972            .session()
1973            .expect("launched")
1974            .agent_token()
1975            .to_string();
1976        let vm = sandbox.microvm().expect("launched").clone();
1977
1978        let detached = sandbox.detach().expect("a RUNNING VM detaches");
1979        assert_eq!(detached.microvm_id, vm.id);
1980        assert_eq!(detached.endpoint, vm.endpoint);
1981        assert_eq!(detached.agent_token(), token);
1982        assert_eq!(detached.region, Region::UsEast1);
1983        let printed = format!("{detached:?} {sandbox:?}");
1984        assert!(!printed.contains(&token), "{printed}");
1985        assert!(printed.contains("detached: true"), "{printed}");
1986
1987        assert!(sandbox.detached());
1988        assert!(
1989            sandbox.drop_warning().is_none(),
1990            "detached is not abandoned"
1991        );
1992        assert!(
1993            sandbox.session().is_none(),
1994            "the session went to the adopter"
1995        );
1996
1997        let before = recorder.calls().len();
1998        for error in [
1999            sandbox.suspend().await.expect_err("detached"),
2000            sandbox.resume().await.map(|_| ()).expect_err("detached"),
2001            sandbox
2002                .run(RunRequest::new().with_image("arn:image"))
2003                .await
2004                .map(|_| ())
2005                .expect_err("detached"),
2006            sandbox
2007                .wait_until_running(Duration::from_secs(1))
2008                .await
2009                .map(|_| ())
2010                .expect_err("detached"),
2011            sandbox.detach().map(|_| ()).expect_err("already detached"),
2012        ] {
2013            assert_eq!(error.kind(), ErrorKind::Precondition, "{error}");
2014            assert!(error.to_string().contains("detached"), "{error}");
2015            assert!(!error.to_string().contains(&token), "{error}");
2016        }
2017        let report = sandbox.terminate(TeardownOpts::default()).await;
2018        assert!(!report.terminate_accepted);
2019        assert!(report.failures[0].contains("detached"), "{report:?}");
2020        assert_eq!(recorder.calls().len(), before, "detached makes no AWS call");
2021        assert!(
2022            sandbox.drop_warning().is_none(),
2023            "a refused terminate is still quiet"
2024        );
2025    }
2026
2027    /// Detach needs a live VM: nothing launched, or already torn down, is refused.
2028    #[tokio::test]
2029    async fn detach_refuses_without_a_live_vm() {
2030        let (mut unlaunched, _recorder, _clock) = planted();
2031        let error = unlaunched.detach().expect_err("nothing launched");
2032        assert_eq!(error.kind(), ErrorKind::Precondition);
2033        assert!(!unlaunched.detached());
2034
2035        let (mut sandbox, recorder, _clock) = launched().await;
2036        recorder
2037            .answer("TerminateMicrovm", Answer::ok("{}"))
2038            .answer(
2039                "GetMicrovm",
2040                Answer::ok(fake::microvm_response("TERMINATED", None)),
2041            );
2042        sandbox.terminate(TeardownOpts::default()).await;
2043        let error = sandbox.detach().expect_err("torn down");
2044        assert_eq!(error.kind(), ErrorKind::Precondition, "{error}");
2045        assert!(!sandbox.detached());
2046    }
2047
2048    /// Drives a launched sandbox to SUSPENDED, which is where resume starts.
2049    ///
2050    /// Both `GetMicrovm` answers are queued **before** the launch rather than one before
2051    /// each wait, and that is not a style choice. The recorder repeats its last queued
2052    /// answer, so a queue left at `[RUNNING]` after the launch and then appended to becomes
2053    /// `[RUNNING, SUSPENDED]` — the suspend wait pops the stale RUNNING, sleeps a poll
2054    /// interval, and the fake's `sleep` **advances the clock**. That five seconds is
2055    /// invisible until it lands in the middle of the STATE-12 window arithmetic, which is
2056    /// exactly where it first showed up. Queueing both up front means every wait matches on
2057    /// its first poll and no test clock moves except when a test moves it.
2058    async fn suspended_with_window(
2059        suspended_sec: u32,
2060    ) -> (Sandbox, Arc<FakeControlPlane>, Arc<TestClock>) {
2061        let (mut sandbox, recorder, clock) = planted();
2062        recorder
2063            .answer(
2064                "RunMicrovm",
2065                Answer::ok(fake::microvm_response("PENDING", None)),
2066            )
2067            .answer(
2068                "GetMicrovm",
2069                Answer::ok(fake::microvm_response("RUNNING", None)),
2070            )
2071            .answer(
2072                "GetMicrovm",
2073                Answer::ok(fake::microvm_response("SUSPENDED", None)),
2074            )
2075            .answer(
2076                "CreateMicrovmAuthToken",
2077                Answer::ok(fake::auth_token_response("proxy-token")),
2078            )
2079            .answer("SuspendMicrovm", Answer::ok(fake::empty_response()));
2080
2081        sandbox
2082            .run(
2083                RunRequest::new()
2084                    .with_image("arn:image")
2085                    .with_suspended_sec(suspended_sec),
2086            )
2087            .await
2088            .expect("launches");
2089        sandbox.suspend().await.expect("suspends");
2090        assert_eq!(
2091            clock.now(),
2092            Duration::ZERO,
2093            "no wait may have slept, or the window arithmetic below is measuring the fake's \
2094             poll interval as well as the elapsed suspension"
2095        );
2096        (sandbox, recorder, clock)
2097    }
2098
2099    /// A suspended sandbox at the default ten-minute window.
2100    async fn suspended() -> (Sandbox, Arc<FakeControlPlane>, Arc<TestClock>) {
2101        suspended_with_window(600).await
2102    }
2103
2104    // ── adopt (#196) ─────────────────────────────────────────────────────────
2105
2106    /// A canary agent token, so a test can assert it appears in no `Debug` or error.
2107    const ADOPT_TOKEN: &str = "adopt-canary-token-9f1c";
2108    const ADOPT_ENDPOINT: &str = "https://mvm-abc123.microvm.us-east-1.amazonaws.com";
2109
2110    /// A plane over the recorder, whose `GetMicrovm` answers the caller queues.
2111    fn adopt_plane() -> (ControlPlane, Arc<FakeControlPlane>, Arc<TestClock>) {
2112        let recorder = Arc::new(FakeControlPlane::new());
2113        let clock = Arc::new(TestClock::new());
2114        let plane = ControlPlane::with_transport(
2115            Arc::clone(&recorder) as Arc<dyn crate::control::transport::Transport>,
2116            Region::UsEast1,
2117            Arc::clone(&clock) as Arc<dyn crate::control::Clock>,
2118        );
2119        (plane, recorder, clock)
2120    }
2121
2122    async fn adopted_in(state: &str) -> (Sandbox, Arc<FakeControlPlane>, Arc<TestClock>) {
2123        let (plane, recorder, clock) = adopt_plane();
2124        recorder.answer(
2125            "GetMicrovm",
2126            Answer::ok(fake::microvm_response(state, None)),
2127        );
2128        let sandbox = Sandbox::adopt(plane, "mvm-abc123", ADOPT_ENDPOINT, ADOPT_TOKEN)
2129            .await
2130            .expect("adopts");
2131        (sandbox, recorder, clock)
2132    }
2133
2134    /// **An adopted sandbox feeds its keepalive.** The idle window is the one `GetMicrovm`
2135    /// reports, and the lifecycle watch starts at the adopted state, so a keepalive on an
2136    /// adopted session paces against the real policy and ends when that sandbox suspends.
2137    ///
2138    /// **Falsification** — 2026-09-24. Assign `sandbox.lifecycle` directly instead of
2139    /// through `set_lifecycle`, or drop the `idle_window` line, and this goes red; restored.
2140    #[tokio::test]
2141    async fn an_adopted_sandbox_reports_the_services_idle_window_and_watches_its_lifecycle() {
2142        let (sandbox, _recorder, _clock) = adopted_in("RUNNING").await;
2143        assert_eq!(sandbox.idle_window(), Some(Duration::from_secs(1800)));
2144        assert_eq!(*sandbox.watch_lifecycle().borrow(), Lifecycle::Running);
2145    }
2146
2147    /// Every service state has a lifecycle, and nothing else parses.
2148    #[test]
2149    fn every_service_state_maps_onto_one_lifecycle() {
2150        for state in crate::constants::MICROVM_STATES {
2151            let lifecycle = Lifecycle::from_service(state).expect("a known state");
2152            assert_eq!(lifecycle.as_str(), state);
2153        }
2154        assert_eq!(Lifecycle::from_service("Running"), None);
2155    }
2156
2157    /// **STATE-3 for an adopted VM.** The lifecycle is the service's, the bootstrap is
2158    /// counted once, and `run` is refused before any wire call, so no run-hook payload is
2159    /// ever sent.
2160    ///
2161    /// **Falsification** — 2026-09-24. Leave the adopted lifecycle at PENDING instead of
2162    /// reading it from `GetMicrovm` and the lifecycle and `token_installed` assertions here
2163    /// go red, as does the suspend in the next test; restored after.
2164    #[tokio::test]
2165    async fn an_adopted_vm_takes_its_lifecycle_from_the_service_and_refuses_run() {
2166        let (mut sandbox, recorder, _) = adopted_in("RUNNING").await;
2167        assert!(sandbox.adopted());
2168        assert_eq!(sandbox.lifecycle(), Lifecycle::Running);
2169        assert!(sandbox.token_installed());
2170        assert_eq!(sandbox.bootstrap_count(), 1);
2171        assert!(sandbox.image_exists());
2172        let session = sandbox.session().expect("a live VM gets a session");
2173        assert_eq!(session.endpoint(), ADOPT_ENDPOINT);
2174        assert_eq!(session.agent_token(), ADOPT_TOKEN);
2175
2176        let error = sandbox
2177            .run(RunRequest::new().with_image("arn:image"))
2178            .await
2179            .expect_err("an adopted VM was bootstrapped by its own launch");
2180        assert_eq!(error.kind(), ErrorKind::InvalidArg);
2181        assert!(error.to_string().contains("STATE-3"), "{error}");
2182        assert_eq!(recorder.call_count("RunMicrovm"), 0);
2183        assert_eq!(recorder.operations(), vec!["GetMicrovm"]);
2184    }
2185
2186    /// A name resolves to the record's VM and adopts it; a missing name or a record from
2187    /// another region is refused before any AWS call.
2188    #[tokio::test]
2189    async fn a_name_adopts_its_records_vm_and_foreign_names_are_refused_locally() {
2190        struct One(crate::names::NameRecord);
2191        impl crate::names::NameStore for One {
2192            fn get(&self, name: &str) -> Result<Option<crate::names::NameRecord>, Error> {
2193                Ok((name == self.0.name).then(|| self.0.clone()))
2194            }
2195            fn put(&self, _: &crate::names::NameRecord) -> Result<(), Error> {
2196                unreachable!()
2197            }
2198            fn delete(&self, _: &str) -> Result<bool, Error> {
2199                unreachable!()
2200            }
2201            fn list(&self) -> Result<Vec<crate::names::NameRecord>, Error> {
2202                Ok(vec![self.0.clone()])
2203            }
2204        }
2205        let record = crate::names::NameRecord::new(
2206            "ci",
2207            "mvm-abc123",
2208            ADOPT_ENDPOINT,
2209            ADOPT_TOKEN,
2210            "us-west-2",
2211        )
2212        .expect("record");
2213        let store = One(record.clone());
2214        let foreign = Sandbox::from_name(&store, "ci", Some(Region::UsEast1), None)
2215            .await
2216            .expect_err("registered in another region");
2217        assert_eq!(foreign.kind(), ErrorKind::InvalidArg);
2218        let missing = Sandbox::from_name(&store, "nope", None, None)
2219            .await
2220            .expect_err("no such name");
2221        assert_eq!(missing.kind(), ErrorKind::Precondition);
2222        assert!(!format!("{foreign} {missing}").contains(ADOPT_TOKEN));
2223
2224        let (plane, recorder, _) = adopt_plane();
2225        recorder.answer(
2226            "GetMicrovm",
2227            Answer::ok(fake::microvm_response("RUNNING", None)),
2228        );
2229        let resolved = crate::names::resolve(&store, "ci", Some(&Region::UsWest2)).expect("ok");
2230        let sandbox = Sandbox::adopt_record(plane, resolved)
2231            .await
2232            .expect("adopts");
2233        assert!(sandbox.adopted());
2234        assert_eq!(
2235            sandbox.microvm().map(|vm| vm.id.as_str()),
2236            Some("mvm-abc123")
2237        );
2238        assert_eq!(
2239            sandbox.session().expect("session").agent_token(),
2240            ADOPT_TOKEN
2241        );
2242        let named = sandbox
2243            .name_record("again")
2244            .expect("an adopted VM can be named");
2245        assert_eq!(
2246            (
2247                named.microvm_id.as_str(),
2248                named.agent_token.as_str(),
2249                named.region.as_str()
2250            ),
2251            ("mvm-abc123", ADOPT_TOKEN, "us-east-1")
2252        );
2253        let unlaunched = Sandbox::with_control_plane(adopt_plane().0);
2254        assert_eq!(
2255            unlaunched
2256                .name_record("x")
2257                .expect_err("nothing to name")
2258                .kind(),
2259            ErrorKind::Precondition
2260        );
2261    }
2262
2263    /// Suspend, resume, and terminate all work through an adopted handle with the usual
2264    /// guards, and nothing about the launch is re-delivered (STATE-7).
2265    #[tokio::test]
2266    async fn an_adopted_vm_suspends_resumes_and_terminates_through_its_handle() {
2267        let (plane, recorder, _) = adopt_plane();
2268        recorder
2269            .answer(
2270                "GetMicrovm",
2271                Answer::ok(fake::microvm_response("RUNNING", None)),
2272            )
2273            .answer(
2274                "GetMicrovm",
2275                Answer::ok(fake::microvm_response("SUSPENDED", None)),
2276            )
2277            .answer(
2278                "GetMicrovm",
2279                Answer::ok(fake::microvm_response("RUNNING", None)),
2280            )
2281            .answer(
2282                "GetMicrovm",
2283                Answer::ok(fake::microvm_response("TERMINATED", None)),
2284            )
2285            .answer("SuspendMicrovm", Answer::ok(fake::empty_response()))
2286            .answer("ResumeMicrovm", Answer::ok(fake::empty_response()))
2287            .answer("TerminateMicrovm", Answer::ok(fake::empty_response()));
2288        let mut sandbox = Sandbox::adopt(plane, "mvm-abc123", ADOPT_ENDPOINT, ADOPT_TOKEN)
2289            .await
2290            .expect("adopts");
2291
2292        sandbox.suspend().await.expect("suspends from RUNNING");
2293        assert_eq!(sandbox.lifecycle(), Lifecycle::Suspended);
2294        sandbox.resume().await.expect("resumes from SUSPENDED");
2295        assert_eq!(sandbox.lifecycle(), Lifecycle::Running);
2296        let report = sandbox
2297            .terminate(TeardownOpts::default().waiting_for_terminated())
2298            .await;
2299        assert!(report.terminate_accepted, "{report:?}");
2300        assert_eq!(sandbox.lifecycle(), Lifecycle::Terminated);
2301        assert!(sandbox.was_terminated());
2302
2303        assert_eq!(recorder.call_count("RunMicrovm"), 0);
2304        assert!(
2305            recorder
2306                .bodies_as_text()
2307                .iter()
2308                .all(|body| !body.contains("runHookPayload") && !body.contains(ADOPT_TOKEN)),
2309            "no launch payload or agent token may reach the control plane"
2310        );
2311        assert_eq!(sandbox.bootstrap_count(), 1, "STATE-3: never above one");
2312    }
2313
2314    /// **STATE-12 for an adopted VM.** With no window from a request of its own, the
2315    /// sandbox refuses a late resume using the `idlePolicy` window `GetMicrovm` reported.
2316    ///
2317    /// **Falsification** — 2026-09-24. Drop the reported-window fallback in
2318    /// `require_open_suspended_window` and this resume reaches `ResumeMicrovm`; restored
2319    /// after.
2320    #[tokio::test]
2321    async fn an_adopted_vm_refuses_a_resume_past_the_reported_window() {
2322        let (plane, recorder, clock) = adopt_plane();
2323        recorder
2324            .answer(
2325                "GetMicrovm",
2326                Answer::ok(fake::microvm_response("RUNNING", None)),
2327            )
2328            .answer(
2329                "GetMicrovm",
2330                Answer::ok(fake::microvm_response("SUSPENDED", None)),
2331            )
2332            .answer("SuspendMicrovm", Answer::ok(fake::empty_response()));
2333        let mut sandbox = Sandbox::adopt(plane, "mvm-abc123", ADOPT_ENDPOINT, ADOPT_TOKEN)
2334            .await
2335            .expect("adopts");
2336        assert_eq!(sandbox.suspended_window(), None, "no request of its own");
2337        sandbox.suspend().await.expect("suspends");
2338
2339        clock.advance(Duration::from_secs(601));
2340        let error = sandbox
2341            .resume()
2342            .await
2343            .expect_err("past the 600 s window the service reported");
2344        assert_eq!(error.kind(), ErrorKind::WindowClosed);
2345        assert_eq!(recorder.call_count("ResumeMicrovm"), 0);
2346    }
2347
2348    /// A VM adopted while already SUSPENDED has no locally observed suspend time, so the
2349    /// resume goes to the service rather than being refused on a guess.
2350    #[tokio::test]
2351    async fn an_adopted_suspended_vm_resumes_through_the_service() {
2352        let (plane, recorder, _) = adopt_plane();
2353        recorder
2354            .answer(
2355                "GetMicrovm",
2356                Answer::ok(fake::microvm_response("SUSPENDED", None)),
2357            )
2358            .answer(
2359                "GetMicrovm",
2360                Answer::ok(fake::microvm_response("RUNNING", None)),
2361            )
2362            .answer("ResumeMicrovm", Answer::ok(fake::empty_response()));
2363        let mut sandbox = Sandbox::adopt(plane, "mvm-abc123", ADOPT_ENDPOINT, ADOPT_TOKEN)
2364            .await
2365            .expect("adopts");
2366        assert_eq!(sandbox.lifecycle(), Lifecycle::Suspended);
2367        let session = sandbox
2368            .resume()
2369            .await
2370            .expect("the service answers the resume");
2371        assert_eq!(session.endpoint(), ADOPT_ENDPOINT);
2372        assert_eq!(recorder.call_count("ResumeMicrovm"), 1);
2373    }
2374
2375    /// **STATE-11 for an adopted VM.** A terminated VM is offered neither a resume nor a
2376    /// suspend, and gets no session.
2377    #[tokio::test]
2378    async fn an_adopted_terminated_vm_refuses_every_transition_locally() {
2379        let (mut sandbox, recorder, _) = adopted_in("TERMINATED").await;
2380        assert!(sandbox.was_terminated());
2381        assert!(sandbox.session().is_none());
2382        let resume = sandbox.resume().await.expect_err("STATE-11");
2383        assert!(resume.to_string().contains("STATE-11"), "{resume}");
2384        let suspend = sandbox.suspend().await.expect_err("STATE-5");
2385        assert!(suspend.to_string().contains("STATE-5"), "{suspend}");
2386        assert_eq!(recorder.operations(), vec!["GetMicrovm"]);
2387    }
2388
2389    /// A VM adopted while PENDING finishes through `wait_until_running`, which counts the
2390    /// bootstrap exactly once.
2391    #[tokio::test]
2392    async fn an_adopted_pending_vm_counts_its_bootstrap_once_on_running() {
2393        let (plane, recorder, _) = adopt_plane();
2394        recorder
2395            .answer(
2396                "GetMicrovm",
2397                Answer::ok(fake::microvm_response("PENDING", None)),
2398            )
2399            .answer(
2400                "GetMicrovm",
2401                Answer::ok(fake::microvm_response("RUNNING", None)),
2402            );
2403        let mut sandbox = Sandbox::adopt(plane, "mvm-abc123", ADOPT_ENDPOINT, ADOPT_TOKEN)
2404            .await
2405            .expect("adopts");
2406        assert_eq!(sandbox.lifecycle(), Lifecycle::Pending);
2407        assert_eq!(sandbox.bootstrap_count(), 0);
2408        sandbox
2409            .wait_until_running(Duration::from_secs(60))
2410            .await
2411            .expect("reaches RUNNING");
2412        assert_eq!(sandbox.bootstrap_count(), 1);
2413        assert!(sandbox.token_installed());
2414        assert!(
2415            sandbox
2416                .run(RunRequest::new().with_image("arn:image"))
2417                .await
2418                .is_err(),
2419            "still one bootstrap per VM"
2420        );
2421        assert_eq!(recorder.call_count("RunMicrovm"), 0);
2422    }
2423
2424    /// The refusals: an empty token before any call, a mismatched endpoint, and a state
2425    /// this client does not know. None of them, and no `Debug`, prints the token.
2426    ///
2427    /// **Falsification** — 2026-09-24. Remove the endpoint comparison and the mismatch case
2428    /// adopts; remove the empty-token check and that case reaches `GetMicrovm`; print the
2429    /// held token in `Debug` and the canary assertion fails. Each restored after.
2430    #[tokio::test]
2431    async fn adopt_refuses_bad_records_and_never_prints_the_token() {
2432        let (plane, recorder, _) = adopt_plane();
2433        let empty = Sandbox::adopt(plane, "mvm-abc123", ADOPT_ENDPOINT, "")
2434            .await
2435            .expect_err("no token");
2436        assert_eq!(empty.kind(), ErrorKind::InvalidArg);
2437        assert_eq!(recorder.calls().len(), 0, "refused before any call");
2438
2439        let (plane, recorder, _) = adopt_plane();
2440        recorder.answer(
2441            "GetMicrovm",
2442            Answer::ok(fake::microvm_response("RUNNING", None)),
2443        );
2444        let mismatch = Sandbox::adopt(plane, "mvm-abc123", "https://other.example", ADOPT_TOKEN)
2445            .await
2446            .expect_err("the endpoint belongs to another record");
2447        assert_eq!(mismatch.kind(), ErrorKind::InvalidArg);
2448        assert!(!mismatch.to_string().contains(ADOPT_TOKEN));
2449
2450        let (plane, recorder, _) = adopt_plane();
2451        recorder.answer(
2452            "GetMicrovm",
2453            Answer::ok(fake::microvm_response("REBOOTING", None)),
2454        );
2455        let unknown = Sandbox::adopt(plane, "mvm-abc123", ADOPT_ENDPOINT, ADOPT_TOKEN)
2456            .await
2457            .expect_err("an unknown state");
2458        assert_eq!(unknown.kind(), ErrorKind::Platform);
2459        assert!(!unknown.to_string().contains(ADOPT_TOKEN));
2460
2461        let (sandbox, _, _) = adopted_in("RUNNING").await;
2462        let printed = format!("{sandbox:?}");
2463        assert!(!printed.contains(ADOPT_TOKEN), "{printed}");
2464        assert!(printed.contains("adopted: true"), "{printed}");
2465    }
2466
2467    /// **STATE-1 and STATE-2.** A launch accepted is PENDING with the image recorded as
2468    /// existing; the platform reporting the run hook succeeded is what marks the token
2469    /// installed.
2470    ///
2471    /// The two halves are asserted with the *same* fake because they are the same call, and
2472    /// what distinguishes them is which fact is recorded when: `image_exists` is true from
2473    /// the accepted launch, `token_installed` only from the RUNNING report. A client that
2474    /// set both at the launch call would pass an end-state assertion and be wrong about a
2475    /// VM that died during startup — which is the next test.
2476    #[tokio::test]
2477    async fn a_launch_records_the_image_and_the_running_report_installs_the_token() {
2478        let (mut sandbox, recorder, _) = planted();
2479        assert_eq!(sandbox.lifecycle(), Lifecycle::Pending);
2480        assert!(
2481            !sandbox.image_exists(),
2482            "nothing is recorded before a launch"
2483        );
2484        assert!(!sandbox.token_installed());
2485
2486        answer_launch(&recorder);
2487        let session = sandbox
2488            .run(RunRequest::new().with_image("arn:image"))
2489            .await
2490            .expect("launches");
2491        assert_eq!(
2492            session.endpoint(),
2493            "https://mvm-abc123.microvm.us-east-1.amazonaws.com"
2494        );
2495
2496        assert_eq!(sandbox.lifecycle(), Lifecycle::Running);
2497        assert!(sandbox.image_exists(), "STATE-1: the image is recorded");
2498        assert!(sandbox.token_installed(), "STATE-2: the token is installed");
2499        assert_eq!(sandbox.bootstrap_count(), 1);
2500        assert_eq!(sandbox.microvm().expect("launched").id, "mvm-abc123");
2501
2502        // The payload really carried the token, read off the wire member rather than from
2503        // the request type — a field on a struct proves nothing about what was emitted.
2504        let body = recorder.first_body("RunMicrovm");
2505        let payload = body["runHookPayload"].as_str().expect("a string");
2506        assert!(payload.starts_with(r#"{"agent_token":"#), "{payload}");
2507        assert_eq!(body["idlePolicy"]["suspendedDurationSeconds"], 600);
2508    }
2509
2510    /// A launch env reaches the wire member the daemon parses, and it is the only thing
2511    /// that changes about the request.
2512    ///
2513    /// Read off `runHookPayload` rather than off the request struct: a field on a struct
2514    /// proves nothing about what was emitted, which is the same argument the test above
2515    /// makes about the token.
2516    #[tokio::test]
2517    async fn a_launch_env_reaches_the_run_hook_payload_on_the_wire() {
2518        let (mut sandbox, recorder, _) = planted();
2519        answer_launch(&recorder);
2520        sandbox
2521            .run(
2522                RunRequest::new()
2523                    .with_image("arn:image")
2524                    .with_launch_env("ANTHROPIC_BASE_URL", "https://gateway.example")
2525                    .with_launch_env("PATH", "/usr/local/bin:/usr/bin:/bin"),
2526            )
2527            .await
2528            .expect("launches");
2529
2530        let body = recorder.first_body("RunMicrovm");
2531        let payload = body["runHookPayload"].as_str().expect("a string");
2532        // One parse deeper, because that is where the daemon reads it from too.
2533        let inner: serde_json::Value =
2534            serde_json::from_str(payload).expect("the payload is itself JSON");
2535        assert!(
2536            inner["agent_token"].as_str().is_some_and(|t| !t.is_empty()),
2537            "the token still rides alongside: {payload}"
2538        );
2539        assert_eq!(
2540            inner["env"]["ANTHROPIC_BASE_URL"],
2541            "https://gateway.example"
2542        );
2543        assert_eq!(inner["env"]["PATH"], "/usr/local/bin:/usr/bin:/bin");
2544    }
2545
2546    /// **The advisory deny reaches the guest's environment, and it changes nothing else
2547    /// about the request.**
2548    ///
2549    /// Read off `runHookPayload` for the reason the launch-env test gives, and the connector
2550    /// member is asserted absent in the same body: `deny_egress` must not quietly ask the
2551    /// platform for anything, because the whole claim is that the platform has nothing to
2552    /// ask for.
2553    ///
2554    /// **Falsification** — 2026-09-13. Make `run` build the payload from
2555    /// `request.launch_env` instead of `request.effective_launch_env()` and the two
2556    /// `https_proxy` assertions go red while every other launch test stays green; restored
2557    /// after.
2558    #[tokio::test]
2559    async fn the_advisory_deny_reaches_the_launch_environment_in_both_spellings() {
2560        let (mut sandbox, recorder, _) = planted();
2561        answer_launch(&recorder);
2562        let request = RunRequest::new().with_image("arn:image").with_deny_egress();
2563        assert_eq!(
2564            request.egress_posture(),
2565            crate::control::EgressPosture::BestEffort
2566        );
2567        sandbox.run(request).await.expect("launches");
2568
2569        let body = recorder.first_body("RunMicrovm");
2570        let inner: serde_json::Value =
2571            serde_json::from_str(body["runHookPayload"].as_str().expect("a string"))
2572                .expect("the payload is itself JSON");
2573        for key in crate::sandbox::DENY_EGRESS_ENV_KEYS {
2574            assert_eq!(
2575                inner["env"][key], DENY_EGRESS_PROXY_URL,
2576                "{key} must point at the black hole: {inner}"
2577            );
2578        }
2579        assert!(
2580            body.get("egressNetworkConnectors").is_none(),
2581            "the advisory deny asks the platform for nothing: {body}"
2582        );
2583    }
2584
2585    /// A caller's own proxy value survives the deny, and a launch that never asked for the
2586    /// deny carries none of the keys.
2587    #[test]
2588    fn the_deny_never_overwrites_a_callers_own_proxy_and_is_absent_by_default() {
2589        let kept = RunRequest::new()
2590            .with_deny_egress()
2591            .with_launch_env("https_proxy", "http://proxy.internal:3128")
2592            .effective_launch_env();
2593        assert_eq!(kept["https_proxy"], "http://proxy.internal:3128");
2594        assert_eq!(kept["http_proxy"], DENY_EGRESS_PROXY_URL);
2595
2596        let plain = RunRequest::new().effective_launch_env();
2597        assert!(
2598            plain.is_empty(),
2599            "a launch that did not ask for the deny sends byte-for-byte what it always sent: \
2600             {plain:?}"
2601        );
2602        assert_eq!(
2603            RunRequest::new().egress_posture(),
2604            crate::control::EgressPosture::Unsealed,
2605            "and its posture is the measured one, not a seal"
2606        );
2607    }
2608
2609    /// **`egress` and `deny_egress` together are refused locally, with zero calls.**
2610    ///
2611    /// The same shape as the over-budget refusal below: a launch carrying both would report
2612    /// `open` while the workload's own tools failed closed, and AWS has no opinion to
2613    /// return about it.
2614    ///
2615    /// **Falsification** — 2026-09-13. Delete the refusal in `run` and this test fails on
2616    /// `expect_err`; the launch then succeeds and reports posture `open`. Restored after.
2617    #[tokio::test]
2618    async fn asking_for_egress_and_the_deny_together_is_refused_before_any_call() {
2619        let (mut sandbox, recorder, _) = planted();
2620        answer_launch(&recorder);
2621        let mut request = RunRequest::new().with_image("arn:image").with_egress();
2622        request.deny_egress = true;
2623
2624        let error = sandbox
2625            .run(request)
2626            .await
2627            .expect_err("opposite intents are not resolved by a precedence rule");
2628        assert_eq!(error.kind(), ErrorKind::InvalidArg);
2629        assert!(error.to_string().contains("opposite things"), "{error}");
2630        assert_eq!(
2631            recorder.calls().len(),
2632            0,
2633            "a locally-refused launch costs nothing"
2634        );
2635    }
2636
2637    /// **A pinned `imageVersion` reaches the wire through the sandbox**, and an unpinned
2638    /// launch omits the member entirely.
2639    ///
2640    /// This is the layer a caller actually uses, so the forwarding is what needs asserting:
2641    /// [`RunRequest::image_version`] is a field on a struct and proves nothing about the
2642    /// emitted body until it is read off the wire member.
2643    ///
2644    /// **Falsification** — run 2026-08-16. Drop `wire.image_version = ...` from `run` and the
2645    /// pinned assertion goes red with the member absent, while every other launch test still
2646    /// passes — which is why this one is here.
2647    #[tokio::test]
2648    async fn a_pinned_image_version_is_forwarded_to_the_launch_request() {
2649        let (mut sandbox, recorder, _) = planted();
2650        answer_launch(&recorder);
2651        sandbox
2652            .run(
2653                RunRequest::new()
2654                    .with_image("arn:image")
2655                    .with_image_version("2.0"),
2656            )
2657            .await
2658            .expect("launches");
2659        assert_eq!(
2660            recorder.first_body("RunMicrovm")["imageVersion"],
2661            "2.0",
2662            "a canary has to launch against the version it means to test"
2663        );
2664
2665        let (mut sandbox, recorder, _) = planted();
2666        answer_launch(&recorder);
2667        sandbox
2668            .run(RunRequest::new().with_image("arn:image"))
2669            .await
2670            .expect("launches");
2671        assert!(
2672            recorder
2673                .first_body("RunMicrovm")
2674                .get("imageVersion")
2675                .is_none(),
2676            "an unpinned launch sends what this client always sent"
2677        );
2678    }
2679
2680    /// **The local refusal, with zero control-plane calls.**
2681    ///
2682    /// The observable difference between this client and one that lets AWS answer, and the
2683    /// same shape as STATE-5's suspend refusal: an over-budget launch env fails before the
2684    /// launch rather than as a `ValidationException` on a member the caller did not know
2685    /// they were filling. botocore does not enforce the ceiling client-side, so without
2686    /// this there is no local signal at all.
2687    ///
2688    /// **Falsification** — build the payload after `run_microvm` instead of before, and the
2689    /// call count assertion goes red.
2690    #[tokio::test]
2691    async fn an_over_budget_launch_env_is_refused_before_any_control_plane_call() {
2692        let (mut sandbox, recorder, _) = planted();
2693        answer_launch(&recorder);
2694
2695        let error = sandbox
2696            .run(
2697                RunRequest::new()
2698                    .with_image("arn:image")
2699                    .with_launch_env("AWS_SESSION_TOKEN", "t".repeat(4096)),
2700            )
2701            .await
2702            .expect_err("a credential-scale launch env does not fit the payload");
2703
2704        assert_eq!(error.kind(), ErrorKind::InvalidArg);
2705        let message = error.to_string();
2706        assert!(message.contains("ceiling of 4096"), "{message}");
2707        assert!(message.contains("launch env contributed"), "{message}");
2708        assert_eq!(
2709            recorder.calls().len(),
2710            0,
2711            "the refusal has to be local: a launch that reached AWS costs a VM and a bill"
2712        );
2713        // And the sandbox is left usable rather than half-launched, so a caller can trim
2714        // the env and retry through the same handle.
2715        assert_eq!(sandbox.lifecycle(), Lifecycle::Pending);
2716        assert_eq!(sandbox.bootstrap_count(), 0);
2717        assert!(sandbox.microvm().is_none());
2718    }
2719
2720    /// **Issue #24's guards reach through `Sandbox`, which is the layer both bindings use.**
2721    ///
2722    /// `microvms-py`'s `run` and `microvms-js`'s `run` both build a [`RunRequest`] and hand it to
2723    /// [`Sandbox::run`], which fills a `RunMicrovmRequest` and calls
2724    /// [`crate::control::ControlPlane::run_microvm`]. So the guards there cover the bindings — but
2725    /// "cover" is a claim about a call chain, and a chain is what a refactor breaks. This asserts
2726    /// it at the layer the bindings actually enter, so a `Sandbox::run` that stopped delegating
2727    /// (or a binding-side default that bypassed the request type) fails a test rather than shipping
2728    /// an unguarded surface to Python and Node.
2729    ///
2730    /// `max_idle_sec` is the case that matters most here, because it is a plain `u32` keyword
2731    /// argument on both bindings with no type narrowing it: `sandbox.run(max_idle_sec=59)` is what
2732    /// a Python caller writes, and before this it reached the wire.
2733    ///
2734    /// **Guard proof.** Delete `require_idle_duration` from `run_microvm` and the idle row goes red
2735    /// on the call count; delete `require_valid_role_arn` and the role row does. Neither guard is in
2736    /// this file, which is the point — the test is about the chain.
2737    #[tokio::test]
2738    async fn the_bindings_layer_refuses_an_illegal_launch_with_zero_calls() {
2739        let cases: [(&str, RunRequest, &str); 3] = [
2740            (
2741                "max_idle_sec=59, which is what a Python keyword argument passes",
2742                RunRequest {
2743                    max_idle_sec: 59,
2744                    ..RunRequest::new().with_image("arn:image")
2745                },
2746                "maxIdleDurationSeconds",
2747            ),
2748            (
2749                "an execution role that is a bare name",
2750                RunRequest {
2751                    execution_role_arn: Some("execution-role".to_string()),
2752                    ..RunRequest::new().with_image("arn:image")
2753                },
2754                "role *name*",
2755            ),
2756            (
2757                "a pinned version with a trailing newline",
2758                RunRequest::new()
2759                    .with_image("arn:image")
2760                    .with_image_version("2.0\n"),
2761                "contains whitespace",
2762            ),
2763        ];
2764
2765        for (label, request, expected) in cases {
2766            let (mut sandbox, recorder, _) = planted();
2767            // The launch answered, so a missing guard is a *successful* launch — a real VM and a
2768            // real bill — rather than a different failure.
2769            answer_launch(&recorder);
2770
2771            // `expect_err` needs the Ok type to be `Debug`, and `run` answers `&mut Session`
2772            // (which is not) — so the discriminant is matched instead. That is also the more
2773            // honest assertion: it says the launch produced no session at all.
2774            let Err(error) = sandbox.run(request).await else {
2775                panic!("{label} must be refused");
2776            };
2777            assert_eq!(error.kind(), ErrorKind::InvalidArg, "{label}");
2778            assert!(
2779                error.to_string().contains(expected),
2780                "{label}: wanted {expected:?}, got {error}"
2781            );
2782            assert_eq!(
2783                recorder.calls().len(),
2784                0,
2785                "{label}: the refusal has to be local, because a launch that reached AWS costs a \
2786                 VM and a bill"
2787            );
2788            assert_eq!(sandbox.lifecycle(), Lifecycle::Pending);
2789            assert!(sandbox.microvm().is_none());
2790        }
2791    }
2792
2793    /// A launch whose VM dies during startup leaves the token **not** installed, because
2794    /// the run hook is what delivers it and a dead VM ran no hook.
2795    ///
2796    /// The distinction this pins is the one the test above cannot: both a correct client and
2797    /// one that installs the token at the launch call reach RUNNING with the token set, and
2798    /// only this case separates them.
2799    #[tokio::test]
2800    async fn a_launch_that_dies_during_startup_installs_no_token() {
2801        let (mut sandbox, recorder, _) = planted();
2802        recorder
2803            .answer(
2804                "RunMicrovm",
2805                Answer::ok(fake::microvm_response("PENDING", None)),
2806            )
2807            .answer(
2808                "GetMicrovm",
2809                Answer::ok(fake::microvm_response(
2810                    "TERMINATED",
2811                    Some("run hook returned 500"),
2812                )),
2813            );
2814
2815        let error = sandbox
2816            .run(RunRequest::new().with_image("arn:image"))
2817            .await
2818            .expect_err("a VM that died during startup is not a launch");
2819        assert_eq!(error.kind(), ErrorKind::LaunchDied);
2820        assert!(
2821            error.to_string().contains("run hook returned 500"),
2822            "{error}"
2823        );
2824
2825        assert!(
2826            !sandbox.token_installed(),
2827            "the run hook never answered, so nothing was installed"
2828        );
2829        assert_eq!(
2830            sandbox.bootstrap_count(),
2831            0,
2832            "a bootstrap counted here would make STATE-3's ceiling unreachable for a retry"
2833        );
2834        assert_eq!(sandbox.lifecycle(), Lifecycle::Pending);
2835        assert!(
2836            sandbox.image_exists(),
2837            "STATE-1 still holds: the launch was accepted against a real image"
2838        );
2839    }
2840
2841    /// **STATE-3, the guard proof.** A second `run` on one sandbox is refused, and it is
2842    /// refused with **zero** further control-plane calls.
2843    ///
2844    /// The call count is the assertion that matters. A client that let the second launch
2845    /// through would create a second VM and silently address two guests through one handle,
2846    /// and it would also deliver a second run-hook payload to a daemon whose one-shot
2847    /// bootstrap answers 409 — so "an error came back" is not enough to tell the two apart.
2848    ///
2849    /// **Falsification** — delete the `bootstrap_count > 0` branch from `run` and this test
2850    /// is red on both the count and the call total. Verified; see the packet's guard proofs.
2851    #[tokio::test]
2852    async fn a_second_run_on_one_sandbox_is_refused_before_any_call() {
2853        let (mut sandbox, recorder, _) = launched().await;
2854        assert_eq!(sandbox.bootstrap_count(), 1);
2855        let before = recorder.calls().len();
2856
2857        let error = sandbox
2858            .run(RunRequest::new().with_image("arn:image"))
2859            .await
2860            .expect_err("the token is installed at most once per VM lifetime");
2861        assert_eq!(error.kind(), ErrorKind::InvalidArg);
2862        assert!(error.to_string().contains("STATE-3"), "{error}");
2863        assert!(
2864            error.to_string().contains("needs a second Sandbox"),
2865            "the remedy has to be nameable: {error}"
2866        );
2867
2868        assert_eq!(
2869            sandbox.bootstrap_count(),
2870            1,
2871            "bootstrap_count <= 1 is the Z3-proved invariant"
2872        );
2873        assert_eq!(
2874            recorder.calls().len(),
2875            before,
2876            "the refusal must cost no control-plane call"
2877        );
2878        assert_eq!(recorder.call_count("RunMicrovm"), 1);
2879    }
2880
2881    /// **STATE-4 and STATE-6.** A suspend from RUNNING moves to SUSPENDING before the wait,
2882    /// and the platform reporting SUSPENDED is what moves it to SUSPENDED.
2883    #[tokio::test]
2884    async fn a_suspend_from_running_passes_through_suspending_to_suspended() {
2885        let (sandbox, recorder, _) = suspended().await;
2886        assert_eq!(sandbox.lifecycle(), Lifecycle::Suspended);
2887        assert!(
2888            sandbox.token_installed(),
2889            "a freeze keeps the guest's memory, so the token survives"
2890        );
2891        assert_eq!(sandbox.bootstrap_count(), 1, "no re-bootstrap on a freeze");
2892
2893        let calls = recorder.calls();
2894        let suspend = calls
2895            .iter()
2896            .find(|call| call.operation == "SuspendMicrovm")
2897            .expect("the suspend went out");
2898        assert_eq!(suspend.method, Method::Post);
2899        assert_eq!(suspend.path, "/2025-09-09/microvms/mvm-abc123/suspend");
2900    }
2901
2902    /// A suspend whose VM dies while suspending reports TERMINATED rather than raising, and
2903    /// records it — which is what stops a later resume.
2904    ///
2905    /// TERMINATED is *wanted* by the suspend wait for a reason: a VM that dies mid-suspend
2906    /// is a state to report, not an exception out of the middle of a teardown.
2907    #[tokio::test]
2908    async fn a_vm_that_dies_while_suspending_is_recorded_rather_than_raised() {
2909        let (mut sandbox, recorder, _) = launched().await;
2910        recorder
2911            .answer("SuspendMicrovm", Answer::ok(fake::empty_response()))
2912            .answer(
2913                "GetMicrovm",
2914                Answer::ok(fake::microvm_response("TERMINATED", Some("idle policy"))),
2915            );
2916
2917        sandbox
2918            .suspend()
2919            .await
2920            .expect("a death mid-suspend is a state, not an error");
2921        assert_eq!(sandbox.lifecycle(), Lifecycle::Terminated);
2922        assert!(
2923            sandbox.was_terminated(),
2924            "STATE-11's precondition is recorded"
2925        );
2926    }
2927
2928    /// A suspend whose wire call fails leaves the sandbox in RUNNING, and a retry works.
2929    ///
2930    /// STATE-4's "accepted" is the wire call succeeding. A client that moved to SUSPENDING
2931    /// before the call would strand a failed call there — a state neither suspend nor
2932    /// resume accepts, so one throttled or dropped request would brick the handle.
2933    /// **Falsification** — move `self.lifecycle = Lifecycle::Suspending` back above the
2934    /// `control.suspend` call and the first assertion here reads SUSPENDING, and the retry
2935    /// is refused by the STATE-5 guard.
2936    #[tokio::test]
2937    async fn a_suspend_whose_call_fails_stays_running_and_can_be_retried() {
2938        let (mut sandbox, recorder, _) = planted();
2939        recorder
2940            .answer(
2941                "RunMicrovm",
2942                Answer::ok(fake::microvm_response("PENDING", None)),
2943            )
2944            .answer(
2945                "GetMicrovm",
2946                Answer::ok(fake::microvm_response("RUNNING", None)),
2947            )
2948            .answer(
2949                "GetMicrovm",
2950                Answer::ok(fake::microvm_response("SUSPENDED", None)),
2951            )
2952            .answer(
2953                "CreateMicrovmAuthToken",
2954                Answer::ok(fake::auth_token_response("proxy-token")),
2955            )
2956            // A 409 rather than a 429 or a transport cut, because those two are retried
2957            // inside `send_with_retry` and this test wants the call to fail once, fast.
2958            .answer("SuspendMicrovm", Answer::failure(409, "ConflictException"))
2959            .answer("SuspendMicrovm", Answer::ok(fake::empty_response()));
2960
2961        sandbox
2962            .run(RunRequest::new().with_image("arn:image"))
2963            .await
2964            .expect("launches");
2965
2966        sandbox
2967            .suspend()
2968            .await
2969            .expect_err("the control plane refused the suspend");
2970        assert_eq!(
2971            sandbox.lifecycle(),
2972            Lifecycle::Running,
2973            "a refused suspend must not move the lifecycle: SUSPENDING accepts neither a \
2974             suspend nor a resume, so recording it here would brick the handle"
2975        );
2976
2977        sandbox.suspend().await.expect("the retry suspends");
2978        assert_eq!(sandbox.lifecycle(), Lifecycle::Suspended);
2979        assert_eq!(recorder.call_count("SuspendMicrovm"), 2);
2980    }
2981
2982    /// **STATE-5, the guard proof.** A suspend from anything but RUNNING is refused with
2983    /// **zero** control-plane calls.
2984    ///
2985    /// Every non-RUNNING state, so a state added to `Lifecycle` later is not silently
2986    /// exempt. The count is the load-bearing assertion: letting the call go and reading the
2987    /// service's answer also produces an error, and that error is about a non-running id
2988    /// rather than about which of two things the caller got wrong.
2989    ///
2990    /// **Falsification** — delete the `lifecycle != Running` branch from `suspend` and the
2991    /// SUSPENDED case emits a `SuspendMicrovm` call, turning the count assertion red.
2992    /// Verified; see the packet's guard proofs.
2993    #[tokio::test]
2994    async fn a_suspend_from_a_non_running_state_reaches_no_control_plane_call() {
2995        // From SUSPENDED — the caller who believes they resumed.
2996        let (mut sandbox, recorder, _) = suspended().await;
2997        let before = recorder.call_count("SuspendMicrovm");
2998
2999        let error = sandbox
3000            .suspend()
3001            .await
3002            .expect_err("a suspend is only issued from RUNNING");
3003        assert_eq!(error.kind(), ErrorKind::InvalidArg);
3004        assert!(error.to_string().contains("STATE-5"), "{error}");
3005        assert!(error.to_string().contains("SUSPENDED"), "{error}");
3006        assert_eq!(
3007            recorder.call_count("SuspendMicrovm"),
3008            before,
3009            "the refusal must be local: no second suspend went to the wire"
3010        );
3011
3012        // And from PENDING, before anything launched at all.
3013        let (mut fresh, fresh_recorder, _) = planted();
3014        let error = fresh.suspend().await.expect_err("nothing to suspend");
3015        assert_eq!(error.kind(), ErrorKind::Precondition);
3016        assert_eq!(fresh_recorder.calls().len(), 0);
3017    }
3018
3019    /// **STATE-7 and STATE-8.** A resume from SUSPENDED returns to RUNNING, re-delivers
3020    /// **nothing**, and drops the cached proxy token.
3021    ///
3022    /// Three assertions, and each one is a different failure mode. The bootstrap count says
3023    /// no run-hook payload was re-delivered — which a daemon's one-shot bootstrap would
3024    /// answer 409 to, reading like a broken VM. The `RunMicrovm` count says no second launch
3025    /// happened. And `is_cached` says the token was invalidated, which is the only
3026    /// observable difference on a path where the endpoint URL does not change: a client that
3027    /// kept the token would produce identical requests until the pre-suspend token stopped
3028    /// validating, and that rejection reads exactly like a dead daemon.
3029    #[tokio::test]
3030    async fn a_resume_reuses_the_token_redelivers_nothing_and_drops_the_proxy_token() {
3031        let (mut sandbox, recorder, _) = suspended().await;
3032
3033        // The session mints once before the suspend, so there is a cached token to drop.
3034        let auth = Arc::clone(
3035            sandbox
3036                .session()
3037                .expect("launched")
3038                .proxy_auth()
3039                .expect("the launch wired a minter"),
3040        );
3041        auth.headers().await.expect("mints");
3042        assert!(
3043            auth.is_cached(),
3044            "there must be a token for the drop to matter"
3045        );
3046        assert_eq!(auth.mint_count(), 1);
3047
3048        recorder
3049            .answer("ResumeMicrovm", Answer::ok(fake::empty_response()))
3050            .answer(
3051                "GetMicrovm",
3052                Answer::ok(fake::microvm_response("RUNNING", None)),
3053            );
3054        let session = sandbox.resume().await.expect("resumes");
3055        assert_eq!(
3056            session.endpoint(),
3057            "https://mvm-abc123.microvm.us-east-1.amazonaws.com"
3058        );
3059
3060        assert_eq!(sandbox.lifecycle(), Lifecycle::Running);
3061        assert!(
3062            sandbox.token_installed(),
3063            "STATE-7: the installed token is reused"
3064        );
3065        assert_eq!(
3066            sandbox.bootstrap_count(),
3067            1,
3068            "STATE-7: a resume re-delivers no run-hook payload"
3069        );
3070        assert_eq!(
3071            recorder.call_count("RunMicrovm"),
3072            1,
3073            "a resume is not a second launch"
3074        );
3075        assert!(
3076            !auth.is_cached(),
3077            "STATE-8: the cached proxy token survived the resume"
3078        );
3079    }
3080
3081    /// **STATE-8's guard proof, on the mint count.** The request after a resume mints a
3082    /// fresh token rather than reusing the pre-suspend one.
3083    ///
3084    /// Separate from the test above because `is_cached` is a fact about the cache and this is
3085    /// a fact about the wire: a client that dropped the cache but kept emitting the old
3086    /// header would pass the first assertion and fail every request.
3087    ///
3088    /// **Falsification** — delete the `proxy.invalidate()` line from `Session::rebind` and
3089    /// the mint count stays at 1 here. Verified; see the packet's guard proofs.
3090    #[tokio::test]
3091    async fn the_request_after_a_resume_mints_a_fresh_proxy_token() {
3092        let (mut sandbox, recorder, _) = suspended().await;
3093        let auth = Arc::clone(
3094            sandbox
3095                .session()
3096                .expect("launched")
3097                .proxy_auth()
3098                .expect("wired"),
3099        );
3100        auth.headers().await.expect("mints");
3101        assert_eq!(auth.mint_count(), 1);
3102
3103        recorder
3104            .answer("ResumeMicrovm", Answer::ok(fake::empty_response()))
3105            .answer(
3106                "GetMicrovm",
3107                Answer::ok(fake::microvm_response("RUNNING", None)),
3108            );
3109        sandbox.resume().await.expect("resumes");
3110
3111        // No clock movement at all, so a re-mint here is caused by the invalidation rather
3112        // than by the refresh window rolling over.
3113        auth.headers().await.expect("re-mints");
3114        assert_eq!(
3115            auth.mint_count(),
3116            2,
3117            "STATE-8: the resume did not force a fresh mint"
3118        );
3119    }
3120
3121    /// **STATE-12, the guard proof.** A resume past the launch-time suspended window is
3122    /// refused locally, with **zero** resume calls, and the message names the window.
3123    ///
3124    /// Four assertions, and the packet asks for all four because each rules out a different
3125    /// wrong implementation. `ErrorKind::WindowClosed` rather than `Timeout` separates this
3126    /// from the client that waits. Zero `ResumeMicrovm` calls is what "before any wire call"
3127    /// means. The elapsed reading rules out an implementation that polled to the deadline
3128    /// first — the fake's clock advances on every `sleep`, so a poll loop here would show up
3129    /// as time passed. And the message naming both numbers plus `idlePolicy` is what sends a
3130    /// reader to the finding rather than looking for the flag that reopens the window.
3131    ///
3132    /// **Falsification** — delete the `require_open_suspended_window` call from `resume` and
3133    /// the fake answers TERMINATED, so this test fails with an `ErrorKind::LaunchDied` after
3134    /// a `ResumeMicrovm` call, naming neither the window nor the seconds elapsed. Verified;
3135    /// see the packet's guard proofs.
3136    /// A keepalive reads this receiver instead of the lock, so every transition must reach
3137    /// it: a suspend it missed would be undone by the keepalive's next poll auto-resuming
3138    /// the VM.
3139    #[tokio::test]
3140    async fn the_lifecycle_watch_follows_every_transition() {
3141        let (mut sandbox, recorder, _clock) = planted();
3142        let watch = sandbox.watch_lifecycle();
3143        assert_eq!(*watch.borrow(), Lifecycle::Pending);
3144        recorder
3145            .answer(
3146                "RunMicrovm",
3147                Answer::ok(fake::microvm_response("PENDING", None)),
3148            )
3149            .answer(
3150                "GetMicrovm",
3151                Answer::ok(fake::microvm_response("RUNNING", None)),
3152            )
3153            .answer(
3154                "GetMicrovm",
3155                Answer::ok(fake::microvm_response("SUSPENDED", None)),
3156            )
3157            .answer(
3158                "CreateMicrovmAuthToken",
3159                Answer::ok(fake::auth_token_response("proxy-token")),
3160            )
3161            .answer("SuspendMicrovm", Answer::ok(fake::empty_response()));
3162        sandbox
3163            .run(RunRequest::new().with_image("arn:image"))
3164            .await
3165            .expect("launches");
3166        assert_eq!(*watch.borrow(), Lifecycle::Running);
3167        sandbox.suspend().await.expect("suspends");
3168        assert_eq!(*watch.borrow(), Lifecycle::Suspended);
3169        assert_eq!(*watch.borrow(), sandbox.lifecycle());
3170    }
3171
3172    #[tokio::test]
3173    async fn a_resume_past_the_suspended_window_is_refused_before_the_wire() {
3174        // A short window, so the arithmetic is legible: sixty seconds asked for at launch.
3175        let (mut sandbox, recorder, clock) = suspended_with_window(60).await;
3176        assert_eq!(sandbox.suspended_window(), Some(Duration::from_secs(60)));
3177        assert_eq!(
3178            sandbox.idle_window(),
3179            Some(Duration::from_secs(600)),
3180            "the launch's own maxIdleDurationSeconds, which a keepalive checks its cadence against"
3181        );
3182
3183        // The window closes. This is the whole of what the clock injection buys.
3184        clock.advance(Duration::from_secs(61));
3185
3186        // What a service says about a VM the idlePolicy already terminated — which is what a
3187        // client without the local check would spend its poll timeout discovering.
3188        recorder
3189            .answer("ResumeMicrovm", Answer::ok(fake::empty_response()))
3190            .answer(
3191                "GetMicrovm",
3192                Answer::ok(fake::microvm_response(
3193                    "TERMINATED",
3194                    Some("suspended window elapsed"),
3195                )),
3196            );
3197        let elapsed_before = clock.now();
3198        let resumes_before = recorder.call_count("ResumeMicrovm");
3199
3200        let error = sandbox
3201            .resume()
3202            .await
3203            .expect_err("the window the launch set has closed");
3204
3205        assert_eq!(error.kind(), ErrorKind::WindowClosed);
3206        assert_eq!(error.code(), "ERR_WINDOW_CLOSED");
3207        let message = error.to_string();
3208        assert!(message.contains("61s"), "the elapsed time: {message}");
3209        assert!(
3210            message.contains("60s suspendedDurationSeconds"),
3211            "the window has to be named: {message}"
3212        );
3213        assert!(message.contains("idlePolicy"), "the finding: {message}");
3214        assert!(
3215            message.contains("Refused before ResumeMicrovm"),
3216            "why the refusal is local: {message}"
3217        );
3218        assert_eq!(
3219            recorder.call_count("ResumeMicrovm"),
3220            resumes_before,
3221            "the refusal must come before ResumeMicrovm"
3222        );
3223        assert_eq!(
3224            clock.now(),
3225            elapsed_before,
3226            "the refusal must be immediate rather than polled to the deadline"
3227        );
3228    }
3229
3230    /// **STATE-12's fallback.** With no window from its own request, the sandbox uses the
3231    /// `suspendedDurationSeconds` that `GetMicrovm` reported (600 in the fake).
3232    #[tokio::test]
3233    async fn the_window_falls_back_to_the_idle_policy_the_service_reported() {
3234        let (mut sandbox, recorder, clock) = suspended_with_window(60).await;
3235        sandbox.suspended_window = None;
3236        recorder.answer("ResumeMicrovm", Answer::ok(fake::empty_response()));
3237        clock.advance(Duration::from_secs(601));
3238        let error = sandbox
3239            .resume()
3240            .await
3241            .expect_err("past the reported window");
3242        assert_eq!(error.kind(), ErrorKind::WindowClosed);
3243        assert!(error.to_string().contains("600s"), "{error}");
3244        assert_eq!(recorder.call_count("ResumeMicrovm"), 0);
3245    }
3246
3247    /// **#195 at the sandbox.** A client-token launch that adopts an idle-suspended VM is
3248    /// resumed and reaches RUNNING with the token counted once. The same poll on a launch
3249    /// without a caller token is still a startup death (TRAP-8).
3250    #[tokio::test]
3251    async fn a_client_token_launch_resumes_the_vm_it_adopted() {
3252        let (mut sandbox, recorder, _) = planted();
3253        recorder
3254            .answer(
3255                "RunMicrovm",
3256                Answer::ok(fake::microvm_response("SUSPENDED", None)),
3257            )
3258            .answer(
3259                "GetMicrovm",
3260                Answer::ok(fake::microvm_response("SUSPENDED", None)),
3261            )
3262            .answer(
3263                "GetMicrovm",
3264                Answer::ok(fake::microvm_response("RUNNING", None)),
3265            )
3266            .answer("ResumeMicrovm", Answer::ok(fake::empty_response()))
3267            .answer(
3268                "CreateMicrovmAuthToken",
3269                Answer::ok(fake::auth_token_response("proxy-token")),
3270            );
3271        let mut request = RunRequest::new().with_image("arn:image");
3272        request.client_token = Some("job-195".into());
3273        request.agent_token = Some("agent-token".into());
3274        sandbox
3275            .run(request)
3276            .await
3277            .expect("the adopted VM is resumed");
3278        assert_eq!(sandbox.lifecycle(), Lifecycle::Running);
3279        assert_eq!(sandbox.bootstrap_count(), 1);
3280        assert_eq!(recorder.call_count("ResumeMicrovm"), 1);
3281
3282        let (mut fresh, recorder, _) = planted();
3283        recorder
3284            .answer(
3285                "RunMicrovm",
3286                Answer::ok(fake::microvm_response("PENDING", None)),
3287            )
3288            .answer(
3289                "GetMicrovm",
3290                Answer::ok(fake::microvm_response("SUSPENDED", Some("hook failed"))),
3291            );
3292        let error = fresh
3293            .run(RunRequest::new().with_image("arn:image"))
3294            .await
3295            .expect_err("a fresh launch that reads SUSPENDED died during startup");
3296        assert_eq!(error.kind(), ErrorKind::LaunchDied);
3297        assert_eq!(recorder.call_count("ResumeMicrovm"), 0);
3298    }
3299
3300    /// **`wait: false`.** The launch returns once accepted, with an addressable session and
3301    /// the lifecycle still PENDING; `wait_until_running` finishes it, exactly once.
3302    #[tokio::test]
3303    async fn a_launch_that_does_not_wait_is_finished_by_wait_until_running() {
3304        let (mut sandbox, recorder, _) = planted();
3305        answer_launch(&recorder);
3306        let mut request = RunRequest::new().with_image("arn:image");
3307        request.wait = false;
3308        let endpoint = sandbox
3309            .run(request)
3310            .await
3311            .expect("accepted")
3312            .endpoint()
3313            .to_string();
3314        assert!(endpoint.starts_with("https://"), "{endpoint}");
3315        assert_eq!(sandbox.lifecycle(), Lifecycle::Pending);
3316        assert_eq!(sandbox.bootstrap_count(), 0);
3317        assert_eq!(recorder.call_count("GetMicrovm"), 0, "nothing polled yet");
3318
3319        sandbox
3320            .wait_until_running(DEFAULT_READY_TIMEOUT)
3321            .await
3322            .expect("reaches RUNNING");
3323        assert_eq!(sandbox.lifecycle(), Lifecycle::Running);
3324        assert_eq!(sandbox.bootstrap_count(), 1);
3325        let error = sandbox
3326            .wait_until_running(DEFAULT_READY_TIMEOUT)
3327            .await
3328            .expect_err("a running launch has nothing to wait for");
3329        assert_eq!(error.kind(), ErrorKind::InvalidArg);
3330        assert_eq!(sandbox.bootstrap_count(), 1, "counted once (STATE-3)");
3331    }
3332
3333    /// A resume **inside** the window goes through, so the guard is a comparison rather than
3334    /// a blanket refusal of every resume.
3335    ///
3336    /// The boundary is inclusive: elapsed exactly equal to the window is still open, matching
3337    /// the Python's `elapsed <= window`.
3338    #[tokio::test]
3339    async fn a_resume_inside_the_window_is_allowed_and_the_boundary_is_inclusive() {
3340        let (mut sandbox, recorder, clock) = suspended_with_window(60).await;
3341
3342        // Exactly at the window.
3343        clock.advance(Duration::from_secs(60));
3344        recorder
3345            .answer("ResumeMicrovm", Answer::ok(fake::empty_response()))
3346            .answer(
3347                "GetMicrovm",
3348                Answer::ok(fake::microvm_response("RUNNING", None)),
3349            );
3350        sandbox
3351            .resume()
3352            .await
3353            .expect("60s == the 60s window is open");
3354        assert_eq!(sandbox.lifecycle(), Lifecycle::Running);
3355        assert_eq!(recorder.call_count("ResumeMicrovm"), 1);
3356    }
3357
3358    /// A successful resume clears the stamp, so the next cycle's window is measured from the
3359    /// next suspend rather than from the first one.
3360    ///
3361    /// The failure this rules out is subtle and would only show on a second cycle:
3362    /// accumulating every suspension's elapsed time into one total refuses a resume whose own
3363    /// window is wide open. The clock is advanced past the window *between* the two cycles,
3364    /// which is what makes the accumulating implementation fail here.
3365    #[tokio::test]
3366    async fn a_successful_resume_clears_the_stamp_so_a_second_cycle_measures_its_own_window() {
3367        let (mut sandbox, recorder, clock) = planted();
3368        // Every `GetMicrovm` answer for both cycles, in the order the waits consume them:
3369        // the launch's RUNNING, then SUSPENDED / RUNNING twice. Queued up front for the
3370        // reason `suspended_with_window` documents — a stale answer left at the head makes a
3371        // wait poll twice, and the fake's `sleep` moves the very clock this test is measuring.
3372        recorder
3373            .answer(
3374                "RunMicrovm",
3375                Answer::ok(fake::microvm_response("PENDING", None)),
3376            )
3377            .answer(
3378                "CreateMicrovmAuthToken",
3379                Answer::ok(fake::auth_token_response("proxy-token")),
3380            )
3381            .answer("SuspendMicrovm", Answer::ok(fake::empty_response()))
3382            .answer("ResumeMicrovm", Answer::ok(fake::empty_response()))
3383            .answer(
3384                "GetMicrovm",
3385                Answer::ok(fake::microvm_response("RUNNING", None)),
3386            )
3387            .answer(
3388                "GetMicrovm",
3389                Answer::ok(fake::microvm_response("SUSPENDED", None)),
3390            )
3391            .answer(
3392                "GetMicrovm",
3393                Answer::ok(fake::microvm_response("RUNNING", None)),
3394            )
3395            .answer(
3396                "GetMicrovm",
3397                Answer::ok(fake::microvm_response("SUSPENDED", None)),
3398            )
3399            .answer(
3400                "GetMicrovm",
3401                Answer::ok(fake::microvm_response("RUNNING", None)),
3402            );
3403        sandbox
3404            .run(
3405                RunRequest::new()
3406                    .with_image("arn:image")
3407                    .with_suspended_sec(60),
3408            )
3409            .await
3410            .expect("launches");
3411
3412        for cycle in 0..2 {
3413            sandbox
3414                .suspend()
3415                .await
3416                .unwrap_or_else(|error| panic!("cycle {cycle} suspend: {error}"));
3417            assert_eq!(sandbox.lifecycle(), Lifecycle::Suspended, "cycle {cycle}");
3418
3419            // Forty-five seconds each time — inside the sixty-second window on its own, and
3420            // ninety in total. An implementation that never cleared the stamp measures the
3421            // total and refuses the second resume, which is the whole point of the loop.
3422            clock.advance(Duration::from_secs(45));
3423
3424            sandbox.resume().await.unwrap_or_else(|error| {
3425                panic!("cycle {cycle} resume, whose own window is wide open: {error}")
3426            });
3427            assert_eq!(sandbox.lifecycle(), Lifecycle::Running, "cycle {cycle}");
3428            assert_eq!(
3429                sandbox.bootstrap_count(),
3430                1,
3431                "cycle {cycle}: no re-bootstrap"
3432            );
3433        }
3434
3435        assert_eq!(
3436            clock.now(),
3437            Duration::from_secs(90),
3438            "ninety seconds of accumulated suspension against a sixty-second window: this is \
3439             the total an implementation that never cleared the stamp would have compared"
3440        );
3441        assert_eq!(
3442            recorder.call_count("ResumeMicrovm"),
3443            2,
3444            "both cycles resumed"
3445        );
3446    }
3447
3448    /// **STATE-11, the resume-after-terminate case the packet names.** A terminated VM never
3449    /// returns to RUNNING, and the recorder sees **zero** resume wire calls.
3450    ///
3451    /// The zero count is the assertion the packet asks for, and it is the right one: a client
3452    /// that called and read the answer would also fail, with whatever the service says about
3453    /// a terminated id — which is a different statement from "this VM is gone and even a
3454    /// successful call would hand you a different machine".
3455    #[tokio::test]
3456    async fn a_resume_after_terminate_records_zero_resume_calls() {
3457        let (mut sandbox, recorder, _) = launched().await;
3458        recorder.answer("TerminateMicrovm", Answer::ok(fake::empty_response()));
3459        let report = sandbox.terminate(TeardownOpts::default()).await;
3460        assert!(report.terminate_accepted);
3461        assert!(sandbox.was_terminated());
3462
3463        let error = sandbox
3464            .resume()
3465            .await
3466            .expect_err("a terminated VM never returns to RUNNING");
3467        assert_eq!(error.kind(), ErrorKind::InvalidArg);
3468        assert!(error.to_string().contains("STATE-11"), "{error}");
3469        assert_eq!(
3470            recorder.call_count("ResumeMicrovm"),
3471            0,
3472            "no resume may reach the wire once the VM was terminated"
3473        );
3474        assert_ne!(
3475            sandbox.lifecycle(),
3476            Lifecycle::Running,
3477            "the lifecycle must not return to RUNNING"
3478        );
3479    }
3480
3481    /// **STATE-9 and STATE-10.** A terminate accepted is TERMINATING with the VM recorded
3482    /// terminated; the platform reporting termination complete is what marks TERMINATED.
3483    ///
3484    /// The two are separate states rather than one because the default teardown does not
3485    /// wait — so TERMINATING is a state a report really ends in, and calling it TERMINATED
3486    /// would claim an observation nobody made.
3487    #[tokio::test]
3488    async fn a_terminate_records_terminating_and_the_completion_report_marks_terminated() {
3489        let (mut sandbox, recorder, _) = launched().await;
3490        recorder.answer("TerminateMicrovm", Answer::ok(fake::empty_response()));
3491
3492        let report = sandbox.terminate(TeardownOpts::default()).await;
3493        assert!(report.terminate_accepted, "STATE-9: the call was accepted");
3494        assert!(sandbox.was_terminated(), "STATE-9: recorded terminated");
3495        assert_eq!(
3496            report.lifecycle,
3497            Some(Lifecycle::Terminating),
3498            "the default teardown does not wait, so TERMINATED is not claimed"
3499        );
3500        assert!(!report.leaked(), "{report:?}");
3501
3502        // And with the wait asked for, STATE-10 is observed.
3503        let (mut sandbox, recorder, _) = launched().await;
3504        recorder
3505            .answer("TerminateMicrovm", Answer::ok(fake::empty_response()))
3506            .answer(
3507                "GetMicrovm",
3508                Answer::ok(fake::microvm_response("TERMINATED", None)),
3509            );
3510        let report = sandbox
3511            .terminate(TeardownOpts::default().waiting_for_terminated())
3512            .await;
3513        assert_eq!(report.lifecycle, Some(Lifecycle::Terminated));
3514        assert_eq!(sandbox.lifecycle(), Lifecycle::Terminated);
3515    }
3516
3517    /// The teardown order is VM, then image, then the log group **last** — asserted on the
3518    /// recorder's ledger rather than on the report, because the order is a property of what
3519    /// was emitted.
3520    ///
3521    /// The log group is last because the service can recreate a group deleted before its
3522    /// image. This crate cannot delete it at all, so the assertion is that it is *named
3523    /// after* the image work rather than that a delete call went out — see
3524    /// `TeardownReport::undeleted`.
3525    #[tokio::test]
3526    async fn teardown_deletes_the_vm_then_the_image_and_names_the_log_group_last() {
3527        let (mut sandbox, recorder, _) = planted();
3528        recorder
3529            .answer(
3530                "CreateMicrovmImage",
3531                Answer::created(fake::create_image_response("agentd-conformance")),
3532            )
3533            .answer(
3534                "GetMicrovmImage",
3535                Answer::ok(fake::get_image_response("agentd-conformance", "CREATED")),
3536            );
3537        sandbox
3538            .build_image(CreateImageRequest::new(
3539                "agentd-conformance",
3540                b"binary".to_vec(),
3541                "s3://bucket/img.zip",
3542                "arn:aws:iam::123456789012:role/build",
3543            ))
3544            .await
3545            .expect("builds");
3546        answer_launch(&recorder);
3547        sandbox
3548            .run(RunRequest::new())
3549            .await
3550            .expect("launches from the built image");
3551
3552        recorder
3553            .answer("TerminateMicrovm", Answer::ok(fake::empty_response()))
3554            .answer(
3555                "ListMicrovmImageVersions",
3556                Answer::ok(fake::list_versions_response("1")),
3557            )
3558            .answer(
3559                "DeleteMicrovmImage",
3560                Answer::ok(fake::delete_image_response()),
3561            );
3562
3563        let report = sandbox
3564            .terminate(
3565                TeardownOpts::default()
3566                    .deleting_image()
3567                    .deleting_log_group(),
3568            )
3569            .await;
3570        assert!(report.terminate_accepted);
3571        assert_eq!(report.image_deleted, Some(true));
3572
3573        // The ledger, filtered to the three destructive operations.
3574        let order: Vec<&str> = recorder
3575            .operations()
3576            .into_iter()
3577            .filter(|operation| matches!(*operation, "TerminateMicrovm" | "DeleteMicrovmImage"))
3578            .collect();
3579        assert_eq!(
3580            order,
3581            ["TerminateMicrovm", "DeleteMicrovmImage"],
3582            "the VM goes before the image: a terminating VM holds a reference to it"
3583        );
3584
3585        // The log group is named, and named last.
3586        assert_eq!(
3587            report.undeleted,
3588            ["/aws/lambda-microvms/agentd-conformance"],
3589            "{report:?}"
3590        );
3591        assert!(report.leaked());
3592        assert!(
3593            report
3594                .failures
3595                .last()
3596                .expect("a failure")
3597                .contains("log group"),
3598            "the log group's failure has to be the last one recorded: {report:?}"
3599        );
3600        assert!(
3601            report
3602                .failures
3603                .last()
3604                .expect("a failure")
3605                .contains("terraform destroy"),
3606            "the reason it leaks has to be named: {report:?}"
3607        );
3608    }
3609
3610    /// An image delete that fails every attempt lands its identifier in the report rather
3611    /// than raising, so the CLI can emit what was left behind (CLI-6).
3612    ///
3613    /// The identifier rather than a flag is the whole point: a leak nobody can name is a leak
3614    /// nobody can clean up.
3615    #[tokio::test]
3616    async fn a_failing_image_delete_names_what_it_left_behind() {
3617        let (mut sandbox, recorder, _) = planted();
3618        recorder
3619            .answer(
3620                "CreateMicrovmImage",
3621                Answer::created(fake::create_image_response("img")),
3622            )
3623            .answer(
3624                "GetMicrovmImage",
3625                Answer::ok(fake::get_image_response("img", "CREATED")),
3626            );
3627        sandbox
3628            .build_image(CreateImageRequest::new(
3629                "img",
3630                b"binary".to_vec(),
3631                "s3://bucket/img.zip",
3632                "arn:aws:iam::123456789012:role/build",
3633            ))
3634            .await
3635            .expect("builds");
3636        answer_launch(&recorder);
3637        sandbox.run(RunRequest::new()).await.expect("launches");
3638
3639        recorder
3640            .answer("TerminateMicrovm", Answer::ok(fake::empty_response()))
3641            .answer(
3642                "ListMicrovmImageVersions",
3643                Answer::ok(fake::list_versions_response("1")),
3644            )
3645            .answer(
3646                "DeleteMicrovmImage",
3647                Answer::failure(409, "image is in CREATING"),
3648            );
3649
3650        let mut opts = TeardownOpts::default().deleting_image();
3651        opts.delete_attempts = 3;
3652        opts.delete_backoff = Duration::from_secs(1);
3653        let report = sandbox.terminate(opts).await;
3654
3655        assert_eq!(report.image_deleted, Some(false));
3656        assert_eq!(
3657            report.undeleted,
3658            ["arn:aws:lambda:us-east-1:123456789012:microvm-image:img"],
3659            "{report:?}"
3660        );
3661        assert!(
3662            sandbox.image_exists(),
3663            "the image really is still there, so the flag must say so"
3664        );
3665        assert_eq!(
3666            recorder.call_count("DeleteMicrovmImage"),
3667            3,
3668            "every attempt ran"
3669        );
3670    }
3671
3672    /// A terminate whose own call fails still records the VM as one this client asked to
3673    /// destroy, and names it as undeleted.
3674    ///
3675    /// Recording it is what stops a later resume from being offered against a VM the caller
3676    /// already tried to kill (STATE-11) — the alternative leaves the sandbox looking
3677    /// resumable, which is the worse of the two wrong answers.
3678    #[tokio::test]
3679    async fn a_terminate_whose_call_fails_still_records_the_intent_and_the_leak() {
3680        let (mut sandbox, recorder, _) = launched().await;
3681        recorder.answer(
3682            "TerminateMicrovm",
3683            Answer::failure(500, "InternalServerException"),
3684        );
3685
3686        let report = sandbox.terminate(TeardownOpts::default()).await;
3687        assert!(!report.terminate_accepted);
3688        assert_eq!(report.undeleted, ["mvm-abc123"], "{report:?}");
3689        assert!(sandbox.was_terminated());
3690        assert_eq!(report.lifecycle, Some(Lifecycle::Terminating));
3691        assert!(
3692            !report.failures.is_empty(),
3693            "the swallowed failure is recorded"
3694        );
3695    }
3696
3697    /// A teardown on a sandbox that never launched is a no-op report rather than a panic.
3698    ///
3699    /// The path a caller reaches by tearing down in a `finally` after a launch that failed
3700    /// before any VM existed, which is exactly when a teardown that assumed a VM would turn
3701    /// the real failure into a panic.
3702    #[tokio::test]
3703    async fn a_teardown_before_any_launch_is_an_empty_report() {
3704        let (mut sandbox, recorder, _) = planted();
3705        let report = sandbox.terminate(TeardownOpts::default()).await;
3706        assert!(!report.terminate_accepted);
3707        assert!(!report.leaked());
3708        assert_eq!(report.lifecycle, Some(Lifecycle::Pending));
3709        assert_eq!(recorder.calls().len(), 0);
3710    }
3711
3712    /// A run with no image at all is a precondition error before any call, rather than a
3713    /// launch of an empty identifier.
3714    #[tokio::test]
3715    async fn a_run_with_no_image_is_refused_before_any_call() {
3716        let (mut sandbox, recorder, _) = planted();
3717        let error = sandbox
3718            .run(RunRequest::new())
3719            .await
3720            .expect_err("there is nothing to launch");
3721        assert_eq!(error.kind(), ErrorKind::Precondition);
3722        assert!(error.to_string().contains("build_image first"), "{error}");
3723        assert_eq!(recorder.calls().len(), 0);
3724    }
3725
3726    /// The lifecycle's six states are the symspec's `vm_state` domain, spelled as the service
3727    /// spells them.
3728    ///
3729    /// Asserted rather than assumed because the spelling reaches an error message a reader
3730    /// compares against a `GetMicrovm` response: a lifecycle rendering `Suspended` beside a
3731    /// service saying `SUSPENDED` reads like two different facts.
3732    #[test]
3733    fn the_lifecycle_states_are_the_state_models_domain_in_the_services_spelling() {
3734        let all = [
3735            Lifecycle::Pending,
3736            Lifecycle::Running,
3737            Lifecycle::Suspending,
3738            Lifecycle::Suspended,
3739            Lifecycle::Terminating,
3740            Lifecycle::Terminated,
3741        ];
3742        assert_eq!(
3743            all.map(Lifecycle::as_str),
3744            [
3745                "PENDING",
3746                "RUNNING",
3747                "SUSPENDING",
3748                "SUSPENDED",
3749                "TERMINATING",
3750                "TERMINATED"
3751            ]
3752        );
3753        for state in all {
3754            assert_eq!(state.to_string(), state.as_str());
3755        }
3756        // The four live states are the ones a `Drop` warns about, and TERMINATING is not one
3757        // of them: the platform accepted the terminate, so nobody needs telling.
3758        assert_eq!(
3759            all.iter().filter(|state| state.is_live()).count(),
3760            4,
3761            "a state added to the enum defaults to not-live, which would silence the warning"
3762        );
3763        assert!(!Lifecycle::Terminating.is_live());
3764        assert!(!Lifecycle::Terminated.is_live());
3765    }
3766
3767    /// An agent token is 64 hex characters and never repeats.
3768    ///
3769    /// The distinctness matters because the token is the only thing separating one VM's
3770    /// control API from another's: two VMs sharing a token means either one's caller can
3771    /// drive the other's guest.
3772    #[test]
3773    fn a_minted_agent_token_is_sixty_four_hex_characters_and_never_repeats() {
3774        let minted: std::collections::HashSet<String> =
3775            (0..200).map(|_| mint_agent_token()).collect();
3776        assert_eq!(minted.len(), 200, "a repeated token is a shared guest");
3777        for token in &minted {
3778            assert_eq!(token.len(), 64, "{token}");
3779            assert!(token.chars().all(|c| c.is_ascii_hexdigit()), "{token}");
3780            assert_ne!(
3781                token,
3782                &"0".repeat(64),
3783                "an all-zero draw is a read that did nothing"
3784            );
3785        }
3786    }
3787
3788    /// A sandbox's `Debug` does not print the agent token.
3789    #[tokio::test]
3790    async fn a_sandbox_debug_does_not_print_the_agent_token() {
3791        let (mut sandbox, recorder, _) = planted();
3792        answer_launch(&recorder);
3793        sandbox
3794            .run(
3795                RunRequest::new()
3796                    .with_image("arn:image")
3797                    .with_agent_token("super-secret-agent-token"),
3798            )
3799            .await
3800            .expect("launches");
3801
3802        let rendered = format!("{sandbox:?}");
3803        assert!(rendered.contains("mvm-abc123"), "{rendered}");
3804        assert!(rendered.contains("Running"), "{rendered}");
3805        assert!(
3806            !rendered.contains("super-secret"),
3807            "the agent token reached a Debug string: {rendered}"
3808        );
3809    }
3810
3811    /// A torn-down sandbox does not warn on drop, and an abandoned one is the case the
3812    /// warning exists for.
3813    ///
3814    /// Asserted on the flag rather than on stderr — capturing another thread's stderr from a
3815    /// test is not something this crate can do without a dependency — so what is pinned is
3816    /// the condition the warning branches on. Both halves, so the branch cannot be
3817    /// vacuously true.
3818    #[tokio::test]
3819    async fn a_torn_down_sandbox_is_distinguishable_from_an_abandoned_one() {
3820        let (mut sandbox, recorder, _) = launched().await;
3821        assert!(
3822            sandbox.lifecycle().is_live() && !sandbox.torn_down,
3823            "an abandoned live VM is what the drop warning is for"
3824        );
3825
3826        recorder.answer("TerminateMicrovm", Answer::ok(fake::empty_response()));
3827        sandbox.terminate(TeardownOpts::default()).await;
3828        assert!(
3829            sandbox.torn_down,
3830            "terminate must mark the sandbox torn down"
3831        );
3832        assert!(
3833            !sandbox.lifecycle().is_live(),
3834            "and the lifecycle must no longer read as live"
3835        );
3836    }
3837
3838    #[tokio::test]
3839    async fn stable_launch_key_and_payload_survive_a_fresh_supervisor() {
3840        let mut bodies = Vec::new();
3841        for _ in 0..2 {
3842            let (mut sandbox, recorder, _) = planted();
3843            answer_launch(&recorder);
3844            let mut request = RunRequest::new().with_image("arn:image");
3845            request.client_token = Some("one-unique-job-launch".into());
3846            request.agent_token = Some("persisted-private-secret".into());
3847            sandbox.run(request).await.expect("accepted");
3848            bodies.push(recorder.first_body("RunMicrovm"));
3849        }
3850        assert_eq!(bodies[0]["clientToken"], "one-unique-job-launch");
3851        assert_eq!(bodies[0], bodies[1]);
3852    }
3853
3854    #[tokio::test]
3855    async fn unsafe_or_malformed_stable_launch_is_refused_without_calls() {
3856        for (token, secret, identity) in [
3857            ("key".to_string(), None, false),
3858            ("key".to_string(), Some("secret".into()), true),
3859            ("".to_string(), Some("secret".into()), false),
3860            ("x".repeat(129), Some("secret".into()), false),
3861            ("line\nbreak".to_string(), Some("secret".into()), false),
3862        ] {
3863            let (mut sandbox, recorder, _) = planted();
3864            let mut request = RunRequest::new().with_image("arn:image");
3865            request.client_token = Some(token);
3866            request.agent_token = secret;
3867            request.identity = identity;
3868            let error = sandbox.run(request).await.expect_err("invalid");
3869            assert_eq!(error.kind(), ErrorKind::InvalidArg);
3870            assert!(recorder.calls().is_empty());
3871        }
3872    }
3873}