mock_upcloud/

faults.rs

//! **The fault surface: every measured lie, armable on its own.**
//!
//! Two ways to arm. By NAME, in which case the fault fires every time it can —
//! that is the mode a regression test uses, because a test that only sometimes
//! provokes the defect is a test that only sometimes passes. Or from a SEED, in
//! which case the whole run's weather is derived from that one number: which
//! faults are armed, at what rate, and for how long. A storm prints the seed of
//! every distinct failure, and the seed alone replays it.
//!
//! # Sticky faults
//!
//! [`Fault::OutOfStock`] and [`Fault::RevokedCredential`] are STICKY: once they
//! fire they keep firing until disarmed. That is what was measured — on
//! SCALEWAY, not UpCloud (both are kept as generic cloud-provider faults):
//! Scaleway's `fr-par-1` refused `poweron` with `412 out_of_stock` for days,
//! and a revoked Scaleway key answered 0 servers from the moment it was revoked. A fault that fires
//! once and clears would let a retry loop paper over both, which is precisely
//! the bug that shipped.

use crate::rng::SplitMix64;
use std::collections::BTreeSet;
use std::sync::Mutex;

/// One measured behaviour, armable.
///
/// The variants are the mock's whole vocabulary of wrongness: if a defect cannot
/// be named here it cannot be provoked, and the answer is a new variant, not a
/// flag on a handler.
#[derive(Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Debug)]
pub enum Fault {
    /// **Behaviour 2.** `GET /1.3/price` drops the socket instead of answering.
    /// Not a status: the connection is closed with no reply at all, which is
    /// what made monetize's start-time probe report "the credential in
    /// UPCLOUD_TOKEN could not be verified" for what was a transport failure.
    /// A mock that answered 503 here would never have shown that.
    PriceTransportReset,
    /// **Behaviours 3 and 37.** `412 out_of_stock`, at BOTH doors that need
    /// capacity: `POST /1.3/server/{uuid}/start` (behaviour 3) and
    /// `POST /1.3/server` (behaviour 37). Sticky.
    ///
    /// One variant, not two, because what is out of stock is a PLAN IN A ZONE
    /// and both calls ask for one. A mock that could be sold out at the
    /// poweron and in stock at the create would let a caller pass by knocking
    /// on the luckier door.
    ///
    /// **Verified against Scaleway, not UpCloud; kept as a generic
    /// cloud-provider fault** (ledger X17, owner ruling 2026-09-21). The only
    /// dated measurement of "412 out_of_stock on poweron, for days" is
    /// Scaleway's fr-par-1 from 2026-09-09. UpCloud documents its capacity
    /// refusal as `409 STORAGE_RESOURCES_UNAVAILABLE`. Relabelled, not removed:
    /// it stays in the seeded weather, because removing it re-maps every seed.
    OutOfStock,
    /// **Behaviour 5.** The credential is revoked: lists answer 200 with ZERO
    /// rows, details answer 403. Never a clean 401 anywhere. Sticky.
    ///
    /// **Verified against Scaleway, not UpCloud; kept as a generic
    /// cloud-provider fault** (ledger X17, owner ruling 2026-09-21). The only
    /// dated measurement of this shape (2026-08-26) is the Scaleway IAM change
    /// ("0 servers, 403 on VPC"). A dead UpCloud token is MEASURED as a clean
    /// 401: that is [`Fault::DeadToken`]. Relabelled, not removed: it stays in
    /// the seeded weather, because removing it re-maps every seed.
    RevokedCredential,
    /// **Behaviour 14.** The VNC port reported after a stop/start is the one
    /// from before it, and stays wrong until `remote_access_enabled` is toggled
    /// no→yes. Default ON: it is not an exceptional condition, it is what the
    /// API does.
    StaleVncPort,
    /// **Behaviour 17.** A re-image interrupted mid-install leaves the CD first
    /// in the boot order, so the next boot runs the installer again, forever.
    InstallerLoop,
    /// **A HYPOTHESIS, not a measurement: withhold `created`.**
    ///
    /// This crate had it the other way round and had it WRONG. The header used
    /// to claim the API sends no `created` on these rows, so a young volume and
    /// a six-month-old orphan were the same row and only a title could tell
    /// them apart. That was the brief, and the brief was mistaken.
    ///
    /// MEASURED against the live account 2026-09-20, two volumes, both
    /// endpoints, `created` every time:
    ///
    /// ```text
    /// 018a7b17-5698-4aa9-b160-f6a6daea9921
    ///   GET /1.3/storage/private  → "created": "2026-09-20T10:35:18Z"
    ///   GET /1.3/storage/{uuid}   → "created": "2026-09-20T10:35:18Z"
    /// the Resize Backup's detail  → "created": "2026-09-19T21:54:06Z"
    /// ```
    ///
    /// So the field is sent, the mock now sends it, and this fault only takes
    /// it away on request. It is the same failure this crate has already made
    /// once, in its own words: a mock stricter than the provider is worse than
    /// one that is laxer, because a lax mock misses a bug and a strict one
    /// INVENTS them — the attach refusal manufactured four hundred failures in
    /// one storm, and this manufactured a verdict path the account cannot
    /// produce.
    ///
    /// Kept, at rate 0 and never in seeded weather, for the same reason
    /// [`Fault::CloneSyncsLikeImport`] is kept: the disagreement it provokes between
    /// `classify` and `refine` in `private-gunnar-ops` is real and worth a
    /// regression test, and a hypothesis must be asked for BY NAME rather than
    /// arrive as weather.
    WithholdCreatedField,
    /// A 503 on any write, the ordinary provider wobble. The one fault that is
    /// not interesting on its own and is here so the retry paths are exercised.
    WriteUnavailable,
    /// **Behaviour 9, sharpened.** The `Resize Backup` a stop-resize-start
    /// leaves behind points its `origin` at a volume that is ALREADY GONE, so a
    /// cleanup that resolves the origin gets `STORAGE_NOT_FOUND` rather than a
    /// parent.
    OrphanResizeBackup,
    /// **A HYPOTHESIS, by name: the `Resize Backup` carries NO labels.**
    ///
    /// MEASURED 2026-09-20 on the live account, the backup a twin resize left
    /// behind DID carry the origin volume's labels (`site=gunnar.rs`,
    /// `repo=private-gunnar-ops`, `role=twin`, `volume=twin`), which is what
    /// the mock copies by default. The owner's brief for F39 says the opposite
    /// — an object UpCloud titled itself carries NEITHER a product title NOR
    /// the estate's labels — and whether the provider copies labels on every
    /// backup kind, or only on the one that was measured, is not known.
    ///
    /// So both are reachable, and the sweep is proven against both: the default
    /// is the measurement, and this fault is the brief. Rate 0, never in seeded
    /// weather, asked for by name — the rule [`Fault::WithholdCreatedField`]
    /// already follows, for the same reason: a mock stricter than the provider
    /// invents verdict paths, so a hypothesis must announce itself.
    ///
    /// What a sweep has left when both the title and the labels are silent is
    /// the backup's `origin`, which names the volume it was taken from. That is
    /// the recognition path this fault exists to exercise.
    ResizeBackupUnlabelled,
    /// **The import that does not finish.** The upload completes, every byte is
    /// accounted for — and then the import object goes `failed` with an
    /// `error_code`/`error_message` pair and the storage lands in `error`
    /// instead of `online`. The caller's poll loop has to tell that apart from
    /// `syncing`, and a loop that only looks for `online` waits its whole budget
    /// for a volume that is never coming.
    ImportFailed,
    /// **The sync that outlasts the budget.** The caller polls a direct upload
    /// with a 1200 s budget; this makes the `syncing` wait 1300 s. A timeout
    /// that has never fired is a timeout nobody has read the handler of.
    SyncExceedsBudget,
    /// **The OLD pessimistic guess about a clone, kept by name only.** A clone
    /// waits in `syncing` for 100–130 s, exactly like an import. This was the
    /// mock's DEFAULT until lane T13 found the measurement it said did not
    /// exist: gunnar `deploy/upcloud/tests/clone_probe.rs`, 2026-09-20 — 728 ms
    /// call, `maintenance` → `online` in 47 s, NO `syncing`. The default is now
    /// the measurement; this is the guess, at rate 0 (ledger X3). It replaces
    /// `clone-skips-sync`, whose optimistic "guess" is now simply the default,
    /// and that name no longer parses — loudly, on purpose.
    CloneSyncsLikeImport,
    /// **The guest reads the hypervisor's UTC clock as local time.** The
    /// hypervisor is right; the GUEST is wrong, because it runs gunnar as PID 1
    /// with no `systemd-timedated` and no `/etc/adjtime` to establish what the
    /// RTC holds. Measured: the appliance was +7 198 668 ms from the front
    /// while the front, on the same hypervisor in the same zone, was +468 ms.
    ///
    /// Armed per-server through [`crate::estate::Estate::create_server`], never
    /// applied to the clock itself — a mock that skewed its own clock would
    /// model the symptom and let a fix that subtracts two hours somewhere pass.
    GuestReadsRtcAsLocalTime,
    /// **Inbound UDP replies are dropped.** DEFAULT ON, because it is the
    /// provider's normal: NTP and DNS-over-UDP simply never answer, which is
    /// why `systemd-timesyncd` is useless up there and why `gunnar-clock` takes
    /// signed time from the front over TCP and from nowhere else. A mock that
    /// let UDP through would let a fix that "just uses NTP" look correct here
    /// and fail there.
    UdpInboundDropped,
    /// **The guest ignores DHCP option 121 and has no route off its own /22.**
    /// It comes up with a good address, answers everything sent TO it, and
    /// cannot reach the front at all — so its clock sync and its boot narration
    /// die OUTBOUND while every inbound probe says the box is healthy. It reads
    /// exactly like a two-hour clock bug and is not one. Measured; fixed in
    /// gunnar `35ac0c3`.
    ///
    /// Armed per-server at create, like the RTC interpretation: it is a fact
    /// about the image, not a coin flipped per packet.
    GuestIgnoresDhcpOption121,
    /// **A different machine is answering on the name.** The `name` path's SSH
    /// host key stops matching the `front` and `direct` paths'. This is the
    /// case the three-path check exists for — a re-image mints a new key every
    /// time, so `REMOTE HOST IDENTIFICATION HAS CHANGED` cannot be treated as
    /// alarming on its own, and the only thing that separates a new machine
    /// from a stolen name is three paths agreeing.
    HijackedName,
    /// A create that succeeds at the provider and times out at the client: the
    /// reply is dropped after the object is committed. The idempotency seam's
    /// reason to exist — a plugin that retries on this and does not search by
    /// label buys twice.
    CommitThenDropReply,
    /// **Behaviour 35 — the firewall of a server that READS FINE answers 403.**
    ///
    /// Behaviour 1 is the firewall endpoint answering `403
    /// ERROR_AUTHENTICATION_FAILED` for a server that is GONE. This is the same
    /// body, byte for byte, for a server that is alive and answers its own
    /// `GET /1.3/server/{uuid}` with 200 — the shape of a credential scoped
    /// without the firewall permission. The two are indistinguishable AT THE
    /// FIREWALL ENDPOINT; only asking the server itself tells them apart, which
    /// is the whole point of a caller that reads both before it decides.
    ///
    /// INFERRED from the provider's permission-scoped API tokens, not measured
    /// on this estate's account: on 2026-09-20 every 403 seen at this endpoint
    /// sat beside a 404 on the server, which is behaviour 1. Rate 0 — a shape
    /// that has not been measured is asked for BY NAME and never arrives as
    /// weather. Sticky, like every other credential property.
    FirewallForbidden,
    /// **Behaviour 38 — the detach answers `200` and the volume is STILL
    /// ATTACHED.**
    ///
    /// The write that reports its own success and did not happen. Not a
    /// refusal, not a timeout, not a slow poll: a clean `200` with the server
    /// object in the reply, and the device still on `GET /1.3/server/{uuid}`
    /// afterwards — so a caller that trusts the status and does not READ BACK
    /// goes on to delete a volume that is attached, or leaves a growth half
    /// done believing it finished.
    ///
    /// The mock could not express this at all before: `Estate::detach` had four
    /// outcomes, three refusals and a removal, and no path that answered `200`
    /// without removing the device. So a sweep or a growth written against this
    /// mock could never meet the shape, and every green it printed was a green
    /// about a mock that could only tell the truth. This is the whole argument
    /// for the fault surface — a defect that cannot be named here cannot be
    /// provoked, and the answer is a variant.
    ///
    /// **Not sticky**, because it is a property of ONE WRITE and not of the
    /// account or the zone: stickiness here would mean no detach could ever
    /// succeed again for the run's lifetime, which hides the read-back-and-
    /// retry path this fault exists to exercise (contrast
    /// [`Fault::OutOfStock`], where the shortage really does outlive the call).
    ///
    /// REPORTED, not measured on this account: rate 0, asked for by name, the
    /// same rule [`Fault::WithholdCreatedField`] and [`Fault::CloneSyncsLikeImport`]
    /// already follow.
    DetachSaysSuccessButStaysAttached,
    /// **Behaviour 36 — `GET /1.3/server/{uuid}` answers 404 for a uuid the
    /// LIST carries.** The account contradicting itself: the list says the
    /// server exists, its own detail says `SERVER_NOT_FOUND`.
    ///
    /// REPORTED 2026-09-20 on t14s: a `plan` against the live account read the
    /// state's server uuid as 404 while `GET /1.3/server` showed gunnar-front,
    /// gunnar-appliance, gunnar-twin, holger-front and njord. Whether the uuid
    /// the list carried was the SAME uuid the state named was not captured
    /// before that box was wiped — so this is a report and not a measurement,
    /// and the more likely reading (a stale state naming a server that was
    /// re-laid under a new uuid) needs no fault at all: it is the ordinary
    /// state machine with a uuid that was never created. Rate 0, by name only,
    /// kept so the caller's "the account contradicts itself" refusal has a path
    /// that reaches it.
    DetailNotFoundForListedServer,
    /// **Behaviour 43 — a DEAD token is a clean `401 AUTHENTICATION_FAILED`,
    /// everywhere.** MEASURED 2026-09-14 (private-gunnar-ops ROTATION §2.3:
    /// `UPCLOUD_TOKEN_HENTOR` and `MONETIZE_UPCLOUD_TOKEN`, both 401). The
    /// "0 rows and 403" shape of [`Fault::RevokedCredential`] is the one whose
    /// only dated measurement is Scaleway's (ledger X17); this is UpCloud's.
    /// Sticky, like every property of a credential. Rate 0: new faults stay out
    /// of the seeded weather so existing seeds keep naming the same runs.
    DeadToken,
    /// **Behaviour 40 — a READ answers `502`.** REPORTED (gunnar `wait.rs`: "the
    /// API occasionally returns 502, so a failed poll is not fatal"). The mock
    /// only had 503-on-write; a poll loop that dies on its first 502 was never
    /// exercised. Rate 0 (see [`Fault::DeadToken`]).
    ReadBadGateway,
    /// **Behaviour 60 — the guest's kernel cannot hot-plug PCI.** Decided per
    /// server at create, like the RTC reading: it is a property of the image.
    /// Such a guest never acks the ACPI `_EJ0`, so a virtio attach or detach on
    /// a STARTED server answers **`511 HOTPLUG_FAILED`** — MEASURED 2026-09-14
    /// on the live appliance (tunnr 6.12.104, no `HOTPLUG_PCI`). A hot-plug
    /// kernel needs `HOTPLUG_PCI(_ACPI/_PCIE/_SHPC)`, `PCIEPORTBUS` and
    /// **`PCI_MSI=y`** (without MSI `_OSC` refuses OS control; measured under
    /// KVM). Disarmed, the guest can hot-plug and the virtio call succeeds,
    /// which is what the Ubuntu template and tunnr ≥ 80c3233 do. Rate 0.
    GuestKernelLacksHotplug,
    /// **Behaviour 56, the stricter reading — a size grow of ANY attached
    /// storage is refused `409 STORAGE_ATTACHED`, "must first be detached".**
    /// UpCloud's docs contradict themselves (DATA-SET-GROWTH-DESIGN §2): one
    /// page says the server must be `stopped`, the other that the storage must
    /// be detached. Both refuse a grow under a RUNNING server, which the mock
    /// now always does; this fault is the page that also refuses it under a
    /// stopped one. Not measured either way: rate 0, by name.
    ResizeRequiresDetach,
    /// **Behaviour 69 — the REBOOTING medium** (ledger L113): a medium composed
    /// WITHOUT `install_then=poweroff` (every korp-installer < 0.1.4; the
    /// medium of the 2026-09-14 loop). Its installer start reads **`started`**
    /// — MEASURED 2026-09-14: `cdrom/eject` answered 200 on a STARTED box
    /// mid-pass — for the whole pass, 900–1100 s (gunnar plan.rs: every
    /// measured pass ≥ 900 s; 1000–1100 s live on 2026-09-08). At its end the
    /// guest REBOOTS: with the CD still loaded and first it installs again
    /// (behaviour 17), otherwise it boots the disk. Disarmed, the medium is the
    /// power-off kind (behaviour 46). A property of the MEDIUM, so asked for by
    /// name: rate 0.
    InstallerReboots,
}

impl Fault {
    pub const ALL: &'static [Fault] = &[
        Fault::PriceTransportReset,
        Fault::OutOfStock,
        Fault::RevokedCredential,
        Fault::StaleVncPort,
        Fault::InstallerLoop,
        Fault::WithholdCreatedField,
        Fault::WriteUnavailable,
        Fault::OrphanResizeBackup,
        Fault::ResizeBackupUnlabelled,
        Fault::CommitThenDropReply,
        Fault::ImportFailed,
        Fault::SyncExceedsBudget,
        Fault::CloneSyncsLikeImport,
        Fault::GuestReadsRtcAsLocalTime,
        Fault::UdpInboundDropped,
        Fault::GuestIgnoresDhcpOption121,
        Fault::HijackedName,
        Fault::FirewallForbidden,
        Fault::DetailNotFoundForListedServer,
        Fault::DetachSaysSuccessButStaysAttached,
        Fault::DeadToken,
        Fault::ReadBadGateway,
        Fault::GuestKernelLacksHotplug,
        Fault::ResizeRequiresDetach,
        Fault::InstallerReboots,
    ];

    pub fn name(self) -> &'static str {
        match self {
            Fault::PriceTransportReset => "price-transport-reset",
            Fault::OutOfStock => "out-of-stock",
            Fault::RevokedCredential => "revoked-credential",
            Fault::StaleVncPort => "stale-vnc-port",
            Fault::InstallerLoop => "installer-loop",
            Fault::WithholdCreatedField => "withhold-created-field",
            Fault::WriteUnavailable => "write-unavailable",
            Fault::OrphanResizeBackup => "orphan-resize-backup",
            Fault::ResizeBackupUnlabelled => "resize-backup-unlabelled",
            Fault::CommitThenDropReply => "commit-then-drop-reply",
            Fault::ImportFailed => "import-failed",
            Fault::SyncExceedsBudget => "sync-exceeds-budget",
            Fault::CloneSyncsLikeImport => "clone-syncs-like-import",
            Fault::GuestReadsRtcAsLocalTime => "guest-reads-rtc-as-local-time",
            Fault::UdpInboundDropped => "udp-inbound-dropped",
            Fault::GuestIgnoresDhcpOption121 => "guest-ignores-dhcp-option-121",
            Fault::HijackedName => "hijacked-name",
            Fault::FirewallForbidden => "firewall-forbidden",
            Fault::DetailNotFoundForListedServer => "detail-404-for-listed-server",
            Fault::DetachSaysSuccessButStaysAttached => "detach-says-success",
            Fault::DeadToken => "dead-token",
            Fault::ReadBadGateway => "read-bad-gateway",
            Fault::GuestKernelLacksHotplug => "guest-kernel-lacks-hotplug",
            Fault::ResizeRequiresDetach => "resize-requires-detach",
            Fault::InstallerReboots => "installer-reboots",
        }
    }

    /// **The one line `--help` prints for this fault.**
    ///
    /// It lives here, beside the variant, and the match is EXHAUSTIVE: a new
    /// variant that forgets its line does not compile. That is the whole
    /// mechanism. The help text used to be a hand-written list in the binary
    /// and it went stale exactly as hand-written lists do — it advertised a
    /// fault called `grant-created-field` that has never existed under that
    /// name (the real one is `withhold-created-field`, and it does the
    /// OPPOSITE), and it named 9 of the 16 faults there were. A person who
    /// read it and typed what it said got `no such fault`.
    pub fn summary(self) -> &'static str {
        match self {
            Fault::PriceTransportReset => "GET /1.3/price drops the socket — a transport failure, not a status",
            Fault::OutOfStock => "412 out_of_stock at BOTH doors: the create and the poweron",
            Fault::RevokedCredential => "0 rows on lists, 403 on details; never a clean 401",
            Fault::StaleVncPort => "the console reported after a stop/start is the one from before it",
            Fault::InstallerLoop => "the CD stays first in the boot order, so the installer runs again",
            Fault::WithholdCreatedField => "the rows carry no `created` (a HYPOTHESIS; the account does send it)",
            Fault::WriteUnavailable => "503 on writes, the ordinary provider wobble",
            Fault::OrphanResizeBackup => "the Resize Backup's `origin` names a volume that is already gone",
            Fault::ResizeBackupUnlabelled => "the Resize Backup carries NO labels (a HYPOTHESIS; the measurement says it does)",
            Fault::CommitThenDropReply => "the object is created and the reply never arrives",
            Fault::ImportFailed => "every byte arrives and the import then goes `failed`, the storage `error`",
            Fault::SyncExceedsBudget => "the post-upload `syncing` outlasts the caller's 1200 s budget",
            Fault::CloneSyncsLikeImport => "a clone waits in `syncing` like an import (the OLD guess; measured: 47 s, no sync)",
            Fault::GuestReadsRtcAsLocalTime => "the guest reads the hypervisor's UTC as local time — two hours, decided at create",
            Fault::UdpInboundDropped => "inbound UDP replies never come back, so NTP and DNS-over-UDP are silent",
            Fault::GuestIgnoresDhcpOption121 => "the guest takes the address and not the routes: outbound dies, inbound looks healthy",
            Fault::HijackedName => "a different machine answers the host key on the NAME path",
            Fault::FirewallForbidden => "a LIVE server's firewall_rule answers 403, the same body a deleted one's does",
            Fault::DetailNotFoundForListedServer => "GET /1.3/server/{uuid} answers 404 for a uuid the LIST carries",
            Fault::DetachSaysSuccessButStaysAttached => "the detach answers 200 and the volume stays attached",
            Fault::DeadToken => "a dead token: a clean 401 AUTHENTICATION_FAILED on every call",
            Fault::ReadBadGateway => "a read answers 502, the gateway wobble a poll must survive",
            Fault::GuestKernelLacksHotplug => "the guest cannot hot-plug: a virtio attach/detach on a running server is 511 HOTPLUG_FAILED",
            Fault::ResizeRequiresDetach => "a grow of ANY attached storage is 409 STORAGE_ATTACHED (the stricter docs page)",
            Fault::InstallerReboots => "the medium REBOOTS after its install: `started` for the 900-1100 s pass, then the CD again or the disk",
        }
    }

    pub fn parse(s: &str) -> Option<Fault> {
        Fault::ALL.iter().copied().find(|f| f.name() == s)
    }

    /// Sticky faults keep firing once they have fired.
    pub fn sticky(self) -> bool {
        matches!(self, Fault::OutOfStock | Fault::RevokedCredential | Fault::FirewallForbidden | Fault::DeadToken)
    }

    /// The rate, in parts per thousand, this fault fires at in seeded weather.
    /// These are not guesses dressed as measurements — they are the rates that
    /// make a 100 000-run storm produce every signature a few hundred times,
    /// which is the only property a storm's rates need. The MEASURED rate of
    /// `out_of_stock` at the provider was, for several days, 1000.
    pub fn seeded_rate_per_mille(self) -> u64 {
        match self {
            Fault::PriceTransportReset => 20,
            Fault::OutOfStock => 8,
            Fault::RevokedCredential => 2,
            // Not a fault so much as the weather: on by default everywhere.
            Fault::StaleVncPort => 1000,
            Fault::InstallerLoop => 15,
            // Never in seeded weather: it is a HYPOTHESIS about the provider,
            // contradicted by measurement, kept only so the verdict paths that
            // would depend on an absent field stay testable.
            Fault::WithholdCreatedField => 0,
            Fault::WriteUnavailable => 30,
            Fault::OrphanResizeBackup => 250,
            // Never in seeded weather: the measurement says the labels ARE copied,
            // and this is the brief that says they are not. Asked for by name.
            Fault::ResizeBackupUnlabelled => 0,
            Fault::CommitThenDropReply => 12,
            Fault::ImportFailed => 10,
            Fault::SyncExceedsBudget => 5,
            // Never in seeded weather: it is a GUESS about the provider, not a
            // fault of it, and a guess must be asked for by name.
            Fault::CloneSyncsLikeImport => 0,
            // An appliance gets it and a front does not, and which one is being
            // created is not something a rate can decide — so it fires whenever
            // it is armed, and the caller arms it for the guest that has the
            // shape. In seeded weather it is on for a quarter of the runs,
            // which is roughly the share of created servers that are
            // appliances rather than fronts or twins.
            Fault::GuestReadsRtcAsLocalTime => 250,
            // Not a fault so much as the weather, like the stale VNC port.
            Fault::UdpInboundDropped => 1000,
            // Like the RTC misreading: a property of the image a server was
            // made from, so roughly the share of created servers that are
            // appliances.
            Fault::GuestIgnoresDhcpOption121 => 250,
            // Rare, and it must be: a name that answers a different key is an
            // attack, not weather. It is in seeded weather at all only so the
            // three-path check is exercised without being asked for.
            Fault::HijackedName => 3,
            // Both never in seeded weather. One is INFERRED from the provider's
            // permission scopes and one is REPORTED off a box that was wiped
            // before the uuids could be compared; neither is a measurement, and
            // a shape that has not been measured is asked for by name.
            Fault::FirewallForbidden => 0,
            Fault::DetailNotFoundForListedServer => 0,
            // REPORTED, not measured here. A write that lies about its own
            // success is the kind of shape a storm should not hand out as
            // weather until one real detach has been caught doing it.
            Fault::DetachSaysSuccessButStaysAttached => 0,
            // Lane T13's four: all by name. A new weather fault would add a
            // draw to every call it is checked on and re-map every existing seed.
            Fault::DeadToken => 0,
            Fault::ReadBadGateway => 0,
            Fault::GuestKernelLacksHotplug => 0,
            Fault::ResizeRequiresDetach => 0,
            // A property of the MEDIUM a run composed, not weather: by name.
            Fault::InstallerReboots => 0,
        }
    }
}

/// The armed set plus the run's generator.
pub struct Faults {
    armed: Mutex<BTreeSet<Fault>>,
    /// Faults that fire probabilistically rather than always, and the stream
    /// that decides. `None` = no seeded weather: armed means always.
    weather: Option<Mutex<SplitMix64>>,
    seed: u64,
    fired: Mutex<BTreeSet<Fault>>,
}

impl Faults {
    /// Nothing armed but the weather that is always true: [`Fault::StaleVncPort`]
    /// is the provider's normal behaviour, not an exception, and a mock that
    /// hid it by default would leave the toggle cure untested.
    pub fn none() -> Faults {
        let mut armed = BTreeSet::new();
        armed.insert(Fault::StaleVncPort);
        // Inbound UDP does not come back. That is not an exceptional condition
        // up there, it is Tuesday — and a mock that hid it by default would let
        // every "just use NTP" fix look correct.
        armed.insert(Fault::UdpInboundDropped);
        Faults {
            armed: Mutex::new(armed),
            weather: None,
            seed: 0,
            fired: Mutex::new(BTreeSet::new()),
        }
    }

    /// Nothing at all, not even the provider's normal. For the one test that
    /// asserts the mock can be quiet.
    pub fn quiet() -> Faults {
        Faults {
            armed: Mutex::new(BTreeSet::new()),
            weather: None,
            seed: 0,
            fired: Mutex::new(BTreeSet::new()),
        }
    }

    /// Weather derived from one number. Every fault whose
    /// [`Fault::seeded_rate_per_mille`] is non-zero is armed, and fires at that
    /// rate off a stream derived from the seed and the fault's own name — so
    /// adding a tenth fault does not renumber the other nine's decisions, and a
    /// seed printed today still replays tomorrow.
    pub fn seeded(seed: u64) -> Faults {
        let armed = Fault::ALL
            .iter()
            .copied()
            .filter(|f| f.seeded_rate_per_mille() > 0)
            .collect();
        Faults {
            armed: Mutex::new(armed),
            weather: Some(Mutex::new(SplitMix64::new(seed))),
            seed,
            fired: Mutex::new(BTreeSet::new()),
        }
    }

    pub fn seed(&self) -> u64 {
        self.seed
    }

    pub fn arm(&self, f: Fault) {
        self.armed.lock().unwrap().insert(f);
    }

    pub fn disarm(&self, f: Fault) {
        self.armed.lock().unwrap().remove(&f);
        self.fired.lock().unwrap().remove(&f);
    }

    pub fn is_armed(&self, f: Fault) -> bool {
        self.armed.lock().unwrap().contains(&f)
    }

    /// Ask whether `f` fires NOW. Deterministic when armed by name; seeded when
    /// the run has weather; sticky faults answer `true` forever once they have
    /// answered it once.
    pub fn fires(&self, f: Fault) -> bool {
        if !self.is_armed(f) {
            return false;
        }
        if f.sticky() && self.fired.lock().unwrap().contains(&f) {
            return true;
        }
        let yes = match &self.weather {
            None => true,
            Some(m) => {
                let mut r = m.lock().unwrap();
                let draw = r.below(1000);
                draw < f.seeded_rate_per_mille()
            }
        };
        if yes && f.sticky() {
            self.fired.lock().unwrap().insert(f);
        }
        yes
    }

    /// Everything that has fired at least once, for a run's report line.
    pub fn fired(&self) -> Vec<Fault> {
        self.fired.lock().unwrap().iter().copied().collect()
    }
}

impl Default for Faults {
    fn default() -> Self {
        Faults::none()
    }
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn armed_by_name_fires_every_time() {
        let f = Faults::none();
        f.arm(Fault::WriteUnavailable);
        for _ in 0..100 {
            assert!(f.fires(Fault::WriteUnavailable));
        }
    }

    #[test]
    fn a_disarmed_fault_never_fires() {
        let f = Faults::quiet();
        for v in Fault::ALL {
            assert!(!f.fires(*v), "{}", v.name());
        }
    }

    /// The default is not "no faults" — the stale VNC port is what the provider
    /// does on every stop/start, and hiding it by default is how the toggle cure
    /// went untested for a month.
    #[test]
    fn stale_vnc_is_the_default_weather() {
        assert!(Faults::none().fires(Fault::StaleVncPort));
    }

    #[test]
    fn out_of_stock_is_sticky() {
        let f = Faults::none();
        f.arm(Fault::OutOfStock);
        assert!(f.fires(Fault::OutOfStock));
        assert!(f.fires(Fault::OutOfStock));
        f.disarm(Fault::OutOfStock);
        assert!(!f.fires(Fault::OutOfStock));
    }

    /// A seed names a run. Two `Faults` built from the same seed must make the
    /// same decisions in the same order, or a printed seed is decoration.
    #[test]
    fn a_seed_replays() {
        let a = Faults::seeded(4242);
        let b = Faults::seeded(4242);
        let mut da = vec![];
        let mut db = vec![];
        for _ in 0..500 {
            da.push(a.fires(Fault::WriteUnavailable));
            db.push(b.fires(Fault::WriteUnavailable));
        }
        assert_eq!(da, db);
        assert!(da.iter().any(|x| *x), "3 % of 500 should fire at least once");
        assert!(da.iter().any(|x| !*x), "3 % of 500 should not fire every time");
    }

    #[test]
    fn every_fault_round_trips_its_name() {
        for f in Fault::ALL {
            assert_eq!(Fault::parse(f.name()), Some(*f));
        }
        assert_eq!(Fault::parse("no-such-fault"), None);
    }
}