pub enum Fault {
Show 25 variants
PriceTransportReset,
OutOfStock,
RevokedCredential,
StaleVncPort,
InstallerLoop,
WithholdCreatedField,
WriteUnavailable,
OrphanResizeBackup,
ResizeBackupUnlabelled,
ImportFailed,
SyncExceedsBudget,
CloneSyncsLikeImport,
GuestReadsRtcAsLocalTime,
UdpInboundDropped,
GuestIgnoresDhcpOption121,
HijackedName,
CommitThenDropReply,
FirewallForbidden,
DetachSaysSuccessButStaysAttached,
DetailNotFoundForListedServer,
DeadToken,
ReadBadGateway,
GuestKernelLacksHotplug,
ResizeRequiresDetach,
InstallerReboots,
}Expand description
One measured behaviour, armable.
The variants are the mock’s whole vocabulary of wrongness: if a defect cannot be named here it cannot be provoked, and the answer is a new variant, not a flag on a handler.
Variants§
PriceTransportReset
Behaviour 2. GET /1.3/price drops the socket instead of answering.
Not a status: the connection is closed with no reply at all, which is
what made monetize’s start-time probe report “the credential in
UPCLOUD_TOKEN could not be verified” for what was a transport failure.
A mock that answered 503 here would never have shown that.
OutOfStock
Behaviours 3 and 37. 412 out_of_stock, at BOTH doors that need
capacity: POST /1.3/server/{uuid}/start (behaviour 3) and
POST /1.3/server (behaviour 37). Sticky.
One variant, not two, because what is out of stock is a PLAN IN A ZONE and both calls ask for one. A mock that could be sold out at the poweron and in stock at the create would let a caller pass by knocking on the luckier door.
Verified against Scaleway, not UpCloud; kept as a generic
cloud-provider fault (ledger X17, owner ruling 2026-09-21). The only
dated measurement of “412 out_of_stock on poweron, for days” is
Scaleway’s fr-par-1 from 2026-09-09. UpCloud documents its capacity
refusal as 409 STORAGE_RESOURCES_UNAVAILABLE. Relabelled, not removed:
it stays in the seeded weather, because removing it re-maps every seed.
RevokedCredential
Behaviour 5. The credential is revoked: lists answer 200 with ZERO rows, details answer 403. Never a clean 401 anywhere. Sticky.
Verified against Scaleway, not UpCloud; kept as a generic
cloud-provider fault (ledger X17, owner ruling 2026-09-21). The only
dated measurement of this shape (2026-08-26) is the Scaleway IAM change
(“0 servers, 403 on VPC”). A dead UpCloud token is MEASURED as a clean
401: that is Fault::DeadToken. Relabelled, not removed: it stays in
the seeded weather, because removing it re-maps every seed.
StaleVncPort
Behaviour 14. The VNC port reported after a stop/start is the one
from before it, and stays wrong until remote_access_enabled is toggled
no→yes. Default ON: it is not an exceptional condition, it is what the
API does.
InstallerLoop
Behaviour 17. A re-image interrupted mid-install leaves the CD first in the boot order, so the next boot runs the installer again, forever.
WithholdCreatedField
A HYPOTHESIS, not a measurement: withhold created.
This crate had it the other way round and had it WRONG. The header used
to claim the API sends no created on these rows, so a young volume and
a six-month-old orphan were the same row and only a title could tell
them apart. That was the brief, and the brief was mistaken.
MEASURED against the live account 2026-09-20, two volumes, both
endpoints, created every time:
018a7b17-5698-4aa9-b160-f6a6daea9921
GET /1.3/storage/private → "created": "2026-09-20T10:35:18Z"
GET /1.3/storage/{uuid} → "created": "2026-09-20T10:35:18Z"
the Resize Backup's detail → "created": "2026-09-19T21:54:06Z"So the field is sent, the mock now sends it, and this fault only takes it away on request. It is the same failure this crate has already made once, in its own words: a mock stricter than the provider is worse than one that is laxer, because a lax mock misses a bug and a strict one INVENTS them — the attach refusal manufactured four hundred failures in one storm, and this manufactured a verdict path the account cannot produce.
Kept, at rate 0 and never in seeded weather, for the same reason
Fault::CloneSyncsLikeImport is kept: the disagreement it provokes between
classify and refine in private-gunnar-ops is real and worth a
regression test, and a hypothesis must be asked for BY NAME rather than
arrive as weather.
A 503 on any write, the ordinary provider wobble. The one fault that is not interesting on its own and is here so the retry paths are exercised.
OrphanResizeBackup
Behaviour 9, sharpened. The Resize Backup a stop-resize-start
leaves behind points its origin at a volume that is ALREADY GONE, so a
cleanup that resolves the origin gets STORAGE_NOT_FOUND rather than a
parent.
ResizeBackupUnlabelled
A HYPOTHESIS, by name: the Resize Backup carries NO labels.
MEASURED 2026-09-20 on the live account, the backup a twin resize left
behind DID carry the origin volume’s labels (site=gunnar.rs,
repo=private-gunnar-ops, role=twin, volume=twin), which is what
the mock copies by default. The owner’s brief for F39 says the opposite
— an object UpCloud titled itself carries NEITHER a product title NOR
the estate’s labels — and whether the provider copies labels on every
backup kind, or only on the one that was measured, is not known.
So both are reachable, and the sweep is proven against both: the default
is the measurement, and this fault is the brief. Rate 0, never in seeded
weather, asked for by name — the rule Fault::WithholdCreatedField
already follows, for the same reason: a mock stricter than the provider
invents verdict paths, so a hypothesis must announce itself.
What a sweep has left when both the title and the labels are silent is
the backup’s origin, which names the volume it was taken from. That is
the recognition path this fault exists to exercise.
ImportFailed
The import that does not finish. The upload completes, every byte is
accounted for — and then the import object goes failed with an
error_code/error_message pair and the storage lands in error
instead of online. The caller’s poll loop has to tell that apart from
syncing, and a loop that only looks for online waits its whole budget
for a volume that is never coming.
SyncExceedsBudget
The sync that outlasts the budget. The caller polls a direct upload
with a 1200 s budget; this makes the syncing wait 1300 s. A timeout
that has never fired is a timeout nobody has read the handler of.
CloneSyncsLikeImport
The OLD pessimistic guess about a clone, kept by name only. A clone
waits in syncing for 100–130 s, exactly like an import. This was the
mock’s DEFAULT until lane T13 found the measurement it said did not
exist: gunnar deploy/upcloud/tests/clone_probe.rs, 2026-09-20 — 728 ms
call, maintenance → online in 47 s, NO syncing. The default is now
the measurement; this is the guess, at rate 0 (ledger X3). It replaces
clone-skips-sync, whose optimistic “guess” is now simply the default,
and that name no longer parses — loudly, on purpose.
GuestReadsRtcAsLocalTime
The guest reads the hypervisor’s UTC clock as local time. The
hypervisor is right; the GUEST is wrong, because it runs gunnar as PID 1
with no systemd-timedated and no /etc/adjtime to establish what the
RTC holds. Measured: the appliance was +7 198 668 ms from the front
while the front, on the same hypervisor in the same zone, was +468 ms.
Armed per-server through crate::estate::Estate::create_server, never
applied to the clock itself — a mock that skewed its own clock would
model the symptom and let a fix that subtracts two hours somewhere pass.
UdpInboundDropped
Inbound UDP replies are dropped. DEFAULT ON, because it is the
provider’s normal: NTP and DNS-over-UDP simply never answer, which is
why systemd-timesyncd is useless up there and why gunnar-clock takes
signed time from the front over TCP and from nowhere else. A mock that
let UDP through would let a fix that “just uses NTP” look correct here
and fail there.
GuestIgnoresDhcpOption121
The guest ignores DHCP option 121 and has no route off its own /22.
It comes up with a good address, answers everything sent TO it, and
cannot reach the front at all — so its clock sync and its boot narration
die OUTBOUND while every inbound probe says the box is healthy. It reads
exactly like a two-hour clock bug and is not one. Measured; fixed in
gunnar 35ac0c3.
Armed per-server at create, like the RTC interpretation: it is a fact about the image, not a coin flipped per packet.
HijackedName
A different machine is answering on the name. The name path’s SSH
host key stops matching the front and direct paths’. This is the
case the three-path check exists for — a re-image mints a new key every
time, so REMOTE HOST IDENTIFICATION HAS CHANGED cannot be treated as
alarming on its own, and the only thing that separates a new machine
from a stolen name is three paths agreeing.
CommitThenDropReply
A create that succeeds at the provider and times out at the client: the reply is dropped after the object is committed. The idempotency seam’s reason to exist — a plugin that retries on this and does not search by label buys twice.
FirewallForbidden
Behaviour 35 — the firewall of a server that READS FINE answers 403.
Behaviour 1 is the firewall endpoint answering 403 ERROR_AUTHENTICATION_FAILED for a server that is GONE. This is the same
body, byte for byte, for a server that is alive and answers its own
GET /1.3/server/{uuid} with 200 — the shape of a credential scoped
without the firewall permission. The two are indistinguishable AT THE
FIREWALL ENDPOINT; only asking the server itself tells them apart, which
is the whole point of a caller that reads both before it decides.
INFERRED from the provider’s permission-scoped API tokens, not measured on this estate’s account: on 2026-09-20 every 403 seen at this endpoint sat beside a 404 on the server, which is behaviour 1. Rate 0 — a shape that has not been measured is asked for BY NAME and never arrives as weather. Sticky, like every other credential property.
DetachSaysSuccessButStaysAttached
Behaviour 38 — the detach answers 200 and the volume is STILL
ATTACHED.
The write that reports its own success and did not happen. Not a
refusal, not a timeout, not a slow poll: a clean 200 with the server
object in the reply, and the device still on GET /1.3/server/{uuid}
afterwards — so a caller that trusts the status and does not READ BACK
goes on to delete a volume that is attached, or leaves a growth half
done believing it finished.
The mock could not express this at all before: Estate::detach had four
outcomes, three refusals and a removal, and no path that answered 200
without removing the device. So a sweep or a growth written against this
mock could never meet the shape, and every green it printed was a green
about a mock that could only tell the truth. This is the whole argument
for the fault surface — a defect that cannot be named here cannot be
provoked, and the answer is a variant.
Not sticky, because it is a property of ONE WRITE and not of the
account or the zone: stickiness here would mean no detach could ever
succeed again for the run’s lifetime, which hides the read-back-and-
retry path this fault exists to exercise (contrast
Fault::OutOfStock, where the shortage really does outlive the call).
REPORTED, not measured on this account: rate 0, asked for by name, the
same rule Fault::WithholdCreatedField and Fault::CloneSyncsLikeImport
already follow.
DetailNotFoundForListedServer
Behaviour 36 — GET /1.3/server/{uuid} answers 404 for a uuid the
LIST carries. The account contradicting itself: the list says the
server exists, its own detail says SERVER_NOT_FOUND.
REPORTED 2026-09-20 on t14s: a plan against the live account read the
state’s server uuid as 404 while GET /1.3/server showed gunnar-front,
gunnar-appliance, gunnar-twin, holger-front and njord. Whether the uuid
the list carried was the SAME uuid the state named was not captured
before that box was wiped — so this is a report and not a measurement,
and the more likely reading (a stale state naming a server that was
re-laid under a new uuid) needs no fault at all: it is the ordinary
state machine with a uuid that was never created. Rate 0, by name only,
kept so the caller’s “the account contradicts itself” refusal has a path
that reaches it.
DeadToken
Behaviour 43 — a DEAD token is a clean 401 AUTHENTICATION_FAILED,
everywhere. MEASURED 2026-09-14 (private-gunnar-ops ROTATION §2.3:
UPCLOUD_TOKEN_HENTOR and MONETIZE_UPCLOUD_TOKEN, both 401). The
“0 rows and 403” shape of Fault::RevokedCredential is the one whose
only dated measurement is Scaleway’s (ledger X17); this is UpCloud’s.
Sticky, like every property of a credential. Rate 0: new faults stay out
of the seeded weather so existing seeds keep naming the same runs.
ReadBadGateway
Behaviour 40 — a READ answers 502. REPORTED (gunnar wait.rs: “the
API occasionally returns 502, so a failed poll is not fatal”). The mock
only had 503-on-write; a poll loop that dies on its first 502 was never
exercised. Rate 0 (see Fault::DeadToken).
GuestKernelLacksHotplug
Behaviour 60 — the guest’s kernel cannot hot-plug PCI. Decided per
server at create, like the RTC reading: it is a property of the image.
Such a guest never acks the ACPI _EJ0, so a virtio attach or detach on
a STARTED server answers 511 HOTPLUG_FAILED — MEASURED 2026-09-14
on the live appliance (tunnr 6.12.104, no HOTPLUG_PCI). A hot-plug
kernel needs HOTPLUG_PCI(_ACPI/_PCIE/_SHPC), PCIEPORTBUS and
PCI_MSI=y (without MSI _OSC refuses OS control; measured under
KVM). Disarmed, the guest can hot-plug and the virtio call succeeds,
which is what the Ubuntu template and tunnr ≥ 80c3233 do. Rate 0.
ResizeRequiresDetach
Behaviour 56, the stricter reading — a size grow of ANY attached
storage is refused 409 STORAGE_ATTACHED, “must first be detached”.
UpCloud’s docs contradict themselves (DATA-SET-GROWTH-DESIGN §2): one
page says the server must be stopped, the other that the storage must
be detached. Both refuse a grow under a RUNNING server, which the mock
now always does; this fault is the page that also refuses it under a
stopped one. Not measured either way: rate 0, by name.
InstallerReboots
Behaviour 69 — the REBOOTING medium (ledger L113): a medium composed
WITHOUT install_then=poweroff (every korp-installer < 0.1.4; the
medium of the 2026-09-14 loop). Its installer start reads started
— MEASURED 2026-09-14: cdrom/eject answered 200 on a STARTED box
mid-pass — for the whole pass, 900–1100 s (gunnar plan.rs: every
measured pass ≥ 900 s; 1000–1100 s live on 2026-09-08). At its end the
guest REBOOTS: with the CD still loaded and first it installs again
(behaviour 17), otherwise it boots the disk. Disarmed, the medium is the
power-off kind (behaviour 46). A property of the MEDIUM, so asked for by
name: rate 0.
Implementations§
Source§impl Fault
impl Fault
pub const ALL: &'static [Fault]
pub fn name(self) -> &'static str
Sourcepub fn summary(self) -> &'static str
pub fn summary(self) -> &'static str
The one line --help prints for this fault.
It lives here, beside the variant, and the match is EXHAUSTIVE: a new
variant that forgets its line does not compile. That is the whole
mechanism. The help text used to be a hand-written list in the binary
and it went stale exactly as hand-written lists do — it advertised a
fault called grant-created-field that has never existed under that
name (the real one is withhold-created-field, and it does the
OPPOSITE), and it named 9 of the 16 faults there were. A person who
read it and typed what it said got no such fault.
pub fn parse(s: &str) -> Option<Fault>
Sourcepub fn seeded_rate_per_mille(self) -> u64
pub fn seeded_rate_per_mille(self) -> u64
The rate, in parts per thousand, this fault fires at in seeded weather.
These are not guesses dressed as measurements — they are the rates that
make a 100 000-run storm produce every signature a few hundred times,
which is the only property a storm’s rates need. The MEASURED rate of
out_of_stock at the provider was, for several days, 1000.