1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
//! **`mock-upcloud` — the UpCloud API 1.3, faithfully wrong.**
//!
//! A real HTTP server that answers UpCloud's 1.3 surface out of a state machine.
//! Point `UPCLOUD_API_BASE` at it and the real `monetize-cloud-impl` upcloud
//! plugin, the real `gunnar/deploy/upcloud` procedure and the real `cargo xtask`
//! estate verbs talk to it without knowing. Nothing here reaches the network.
//!
//! # The point is the lies, not the happy path
//!
//! A mock that answers every call correctly proves nothing, because that is not
//! the provider anybody has. Every behaviour below was MEASURED against the live
//! account or is written down in this estate's history (3, 5 and 37 are the
//! exception: verified against Scaleway, not UpCloud — see below), and each one is armable
//! on its own ([`faults::Fault`]) so a defect can be reproduced on demand instead
//! of waited for.
//!
//! | # | behaviour | where it is modelled |
//! |---|---|---|
//! | 1 | a firewall-rule read for a DELETED server answers `403 ERROR_AUTHENTICATION_FAILED`, not 404 | [`http::firewall_rule`], always on — it is not a fault, it is the API |
//! | 2 | `GET /1.3/price` fails at the TRANSPORT level, not with a status | [`Fault::PriceTransportReset`] — the socket is closed mid-reply |
//! | 3 | `412 out_of_stock` on poweron, for days — **verified against Scaleway (fr-par-1 2026-09-09), not UpCloud; kept as a generic cloud-provider fault** (UpCloud documents `409 STORAGE_RESOURCES_UNAVAILABLE`) | [`Fault::OutOfStock`], sticky until disarmed |
//! | 4 | the server LIST carries no `storage_devices`; the storage list DOES carry `labels` | [`render`], always on |
//! | 5 | a revoked credential answers 0 servers and 403 on some endpoints, never a clean 401 — **verified against Scaleway (IAM change 2026-08-26), not UpCloud; kept as a generic cloud-provider fault** (a dead UpCloud token is a clean 401: 43) | [`Fault::RevokedCredential`] |
//! | 6 | create a server: 98–105 s | [`clock::Timings::server_create`] |
//! | 7 | delete front/twin ≈ 60 s; delete an appliance carrying 4 member volumes > 5 min in `maintenance` | [`clock::Timings::server_delete`] |
//! | 8 | the guest's own install ≈ 10 s (`INSTALL-OK` 10 011 ms measured under KVM) | [`clock::Timings::guest_install`] / the KVM half |
//! | 9 | `stop-resize-start` leaves an auto-titled `Resize Backup`, detached, labelled, WITHOUT "gunnar" in the title | [`estate::Estate::modify_server`] |
//! | 10 | ~~young objects are indistinguishable from orphans without `created`~~ **WRONG, and corrected below** | `created` is SENT, on both the list and the detail; [`Fault::WithholdCreatedField`] takes it away on request |
//! | 11 | a server must be stopped (hard) before it can be deleted with its storages | [`estate::Estate::delete_server`] |
//! | 12 | utility IPs are reassigned between re-lays; public IPs change every time | [`estate::Addresses`], shuffled per lay from the seed |
//! | 13 | no serial console — VNC only; a PID-1 panic is invisible except on the framebuffer | the KVM half boots `-serial none` |
//! | 14 | the VNC port goes stale on every stop/start and the API keeps reporting the old one | [`Fault::StaleVncPort`] (default ON — it is the provider's normal) |
//! | 15 | SeaBIOS, no UEFI | the KVM half refuses an OVMF spec by name |
//! | 16 | `cdrom/eject` works on a STARTED server; IDE/virtio detach does not (the virtio half is the guest's — see 60) | [`estate::Estate::eject`] / [`estate::Estate::detach`] |
//! | 17 | a re-image killed mid-install loops the installer (CD stays first in the boot order) | [`estate::Guest::boot`] |
//! | 18 | inbound UDP replies are dropped (DNS/NTP) | [`Fault::UdpInboundDropped`], default ON; [`guest_clock::ntp_answer`] answers silence |
//! | 19 | a guest boots one TIMEZONE OFFSET ahead — +2 h in summer, +1 h in winter — and not every guest does | [`guest_clock`]: the hypervisor's RTC is correct UTC, the GUEST misreads it ([`Fault::GuestReadsRtcAsLocalTime`]) |
//! | 20 | the installer PID 1 brings no network up, so nothing narrates the install window | the KVM half's install image |
//! | 21 | on first boot `unattended-upgrade-shutdown --wait-for-signal` holds `dpkg lock-frontend` for the life of the boot | the Ubuntu template guest |
//! | 22 | a volume NEVER shrinks; a growth is permanent and billed forever | [`estate::Estate::modify_storage`] refuses a shrink by name |
//! | 23 | the direct-upload import and the storage have **two different clocks**: the import says `completed` five seconds in, the storage sits in `syncing` for 100–130 s more | [`estate::Estate::upload`] → [`estate::After::BeginSync`] |
//! | 24 | the import object carries `read_bytes`/`written_bytes` and REAL `md5sum`/`sha256sum` — the ladder verifies the sha256 against the local file | [`digest`], which computes them rather than inventing them |
//! | 25 | the same 43 MB medium is uploaded TWICE per re-image (cdrom, then virtio, because `korp-installer` probes virtio alone) — two imports, two syncs, ~4 min of pure waiting | [`clock::Timings::storage_sync_lo_ms`] |
//! | 26 | `POST /storage/{uuid}/clone` is the candidate fix, and whether it skips the sync is **MEASURED after all — see 53** | [`estate::Estate::clone_storage`]; pessimistic by default, [`Fault::CloneSyncsLikeImport`] for the optimistic guess |
//! | 27 | one wrong clock is SIX red rows: skew-refusal → no-quorum → clock-unchanged → console-key → banner → monetize-poll | [`guest_clock::cascade`] |
//! | 28 | **the skew window is never widened**: a row that needs a wider window is a RED | [`guest_clock::widening_the_window_still_leaves_a_red`] — no number turns a wrong clock green |
//! | 29 | **DHCP option 121 is offered and a guest that ignores it has NO ROUTE off its /22** — outbound dies, inbound stays healthy | [`net::outbound_reach`] / [`Fault::GuestIgnoresDhcpOption121`] |
//! | 30 | **hairpin NAT does not exist**: the box holding a DNAT cannot use it on its own public address | [`net::Reach::NoHairpin`] — and it reads `connection refused` |
//! | 31 | the console is a HOST and a port, and the toggle re-provisions BOTH (`se-sto1.vnc.upcloud.com:60031`) | [`estate::Server::console`], [`render::server`] |
//! | 32 | the guest's install is 2–5 s on the UART while the ladder's wall measurement of the same re-image is 110–253 s | [`clock::Timings::reimage_observers`] — two observers, two orders of magnitude |
//! | 33 | a re-image mints a NEW SSH host key every time; three paths must agree to tell that from a hijacked name | [`estate::Estate::host_key_via`] / [`Fault::HijackedName`] |
//! | 34 | **the door that actually leaks**: `POST /storage/{uuid}/resize` (the twin's xfs growth) takes a `Resize Backup` FIRST, the size of the volume, and hands it back as `resize_backup`; nothing deletes it. Labelled by default (measured); [`Fault::ResizeBackupUnlabelled`] is the brief's hypothesis that it carries nothing at all | [`estate::Estate::resize_filesystem`] |
//! | 35 | the firewall of a server that READS FINE answers the same `403 ERROR_AUTHENTICATION_FAILED` as behaviour 1 — a credential scoped without the firewall permission; only the server's own detail tells "forbidden" from "gone" | [`Fault::FirewallForbidden`], by name only (INFERRED, not measured) |
//! | 36 | `GET /1.3/server/{uuid}` answers `404 SERVER_NOT_FOUND` for a uuid the LIST carries — the account contradicting itself | [`Fault::DetailNotFoundForListedServer`], by name only (REPORTED 2026-09-20, uuids never compared before the box was wiped) |
//! | 37 | a sold-out zone refuses the **create** with the same `412 out_of_stock` it refuses the poweron with, and mints nothing — **verified against Scaleway (fr-par-1 2026-09-09), not UpCloud; kept as a generic cloud-provider fault** | [`estate::Estate::create_server`] / [`Fault::OutOfStock`], one sticky fault for both doors |
//! | 38 | a detach answers `200` **and the volume stays attached** — the write that reports its own success and did not happen; only a read-back tells it from a real one | [`estate::Estate::detach`] / [`Fault::DetachSaysSuccessButStaysAttached`], by name only (REPORTED) |
//! | 39 | an empty path segment (`/1.3//storage`) is a `404` that names no path | [`http`], always on — gunnar `api.rs`, MEASURED |
//! | 40 | a READ answers `502` now and then; a poll must survive it | [`Fault::ReadBadGateway`], by name (REPORTED, gunnar `wait.rs`) |
//! | 41 | `PUT /1.3/server/{uuid}` answers `202` | [`http`], always on — MEASURED 2026-09-19 |
//! | 42 | a chunked PUT to the direct-upload URL answers `411` | [`http`], always on (REPORTED, gunnar `http.rs`) |
//! | 43 | a DEAD token is a clean `401 AUTHENTICATION_FAILED` on every call | [`Fault::DeadToken`] — MEASURED 2026-09-14 (ROTATION §2.3) |
//! | 44 | past the account's MaxIOPS quota (yvra: 10 240 GiB) a create or grow is `403 MAXIOPS_STORAGE_LIMIT_REACHED` | [`estate::Estate::create_storage`] / `modify_storage`, always on (limit MEASURED, code REPORTED) |
//! | 45 | a disk boot reads `started` ~10 s after the start (was 20 s here) | [`clock::Timings::start_ms`] — MEASURED 2026-09-19 |
//! | 46 | **an installer that powers the box off never reads `started`**: `maintenance` t+10…t+130 s, then `stopped` | [`clock::Timings::install_pass_lo_ms`], [`estate::After::GuestPoweredOff`] — MEASURED 2026-09-15/19 |
//! | 47 | `stop_server` with `timeout_action` is a `400` (it belongs to restart) | [`http`], always on — MEASURED 2026-09-14 |
//! | 48 | a plan change is done when the PUT answers (the provider starts at once); a `start` during `maintenance` is `409` | [`estate::Estate::modify_server`] / `start_server` — the provider's order MEASURED 2026-09-21 |
//! | 49 | a `boot_order` PUT does not end an install loop; it is read at the next HYPERVISOR start | always on, now tested — MEASURED 2026-09-14 |
//! | 50 | the VNC toggle settles ~2 s after the "no"; a "yes" inside the window gives the OLD endpoint back | [`clock::Timings::vnc_settle_ms`] — MEASURED (gunnar `console.rs`) |
//! | 51 | the console toggle is `409` while the server is `maintenance` | [`estate::Estate::modify_server`] — MEASURED (memory: upcloud-vnc-port-toggle) |
//! | 52 | an import's storage ends `syncing` → `maintenance` → `online` | [`clock::Timings::sync_tail_maintenance_ms`] — sequence MEASURED, split not |
//! | 53 | **a clone is `maintenance` → `online` in 47 s with NO `syncing`** (26's "not measured" was measured) | [`clock::Timings::clone_online_ms`]; the old guess is [`Fault::CloneSyncsLikeImport`] — MEASURED 2026-09-20 |
//! | 54 | a shrink AND a same-size PUT are `400 SIZE_INVALID` "The new size must be greater than the old size." | [`estate::Estate::modify_storage`] — MEASURED 2026-09-14 (22's code was wrong) |
//! | 55 | a size grow is `maintenance` for 37 s, then `online` | [`clock::Timings::storage_grow_ms`] — MEASURED 2026-09-14 |
//! | 56 | a grow of a storage attached to a RUNNING server is `409 STORAGE_ATTACHED` (terraform 5.44.1 sends it without stopping) | always on; [`Fault::ResizeRequiresDetach`] for the stricter docs page — REPORTED (docs) |
//! | 57 | a storage already attached is `409 STORAGE_ATTACHED` on a second attach | [`estate::Estate::attach_at`] — REPORTED |
//! | 58 | a requested address is honoured, `virtio` takes the first FREE slot, and the guest names virtio disks by slot order, contiguously (vda = virtio:0) | [`estate::Estate::attach_at`], [`estate::Estate::guest_disk_names`], `/mock/disks/{uuid}` |
//! | 59 | a storage in `maintenance`/`syncing`, or the source of a clone still being made, cannot be deleted: `409 STORAGE_STATE_ILLEGAL` | [`estate::Estate::delete_storage`] — REPORTED |
//! | 60 | **virtio hot-plug is the GUEST's**: a hot-plug kernel (needs `PCI_MSI`) takes an attach/detach on a running server; one without answers `511 HOTPLUG_FAILED` (16's `409 VIRTIO_HOTPLUG_UNSUPPORTED` was invented) | [`Fault::GuestKernelLacksHotplug`], per server at create — MEASURED 2026-09-14 |
//! | 61 | an interface that asks for `IPv6` gets IPv6 ("no IPv6" is the estate's law, not UpCloud's) | [`render::server`] — REPORTED |
//! | 62 | **the firewall is evaluated**: off or no rules = open; top-down first match; a drop is silence; the utility net is filtered; UDP replies die behind a catch-all drop | [`net::firewall_admits`], `/mock/inbound/{uuid}`, `/mock/udp-reply/{uuid}` — MEASURED 2026-09-13/14/20 |
//! | 63 | the mock says whom it heard (per credential digest), so a terraform verb can prove the provider talked to THIS mock before `apply`/`destroy` | [`http::Mock::heard_from`], `/mock/heard?token_sha256=` |
//! | 64 | a server PUT changes `hostname`/`title` in place (the mock dropped both; terraform refused its own apply) | [`estate::Estate::rename_server`] — MEASURED 2026-09-21 through 5.44.1 |
//! | 65 | `POST /server/{uuid}/cdrom/load` puts a medium into an EMPTY tray; a loaded one is `409 CDROM_DEVICE_IN_USE` | [`http`] — L110, DOCUMENTED, not measured |
//! | 66 | an import straight after `POST /storage` (no wait) is accepted; any other non-online state is still refused | [`estate::Estate::start_import`], [`estate::After::Created`] — acceptance MEASURED (live receipt 2026-09-21, gunnar `plan.rs:550`); the state it read then is not recorded |
//! | 67 | the direct-upload `PUT` answers the import object BARE (`{"written_bytes": …, "sha256sum": …}`), not wrapped in `storage_import` | [`http`] — MEASURED: the live re-image of 2026-09-21 08:42 read it at the top level, `done: true` |
//! | 68 | **the verified spellings**: a storage `size` is sent as a JSON NUMBER (verified live, monetize grow 2026-09-14), a stop `timeout` as a STRING (verified live, gunnar-upcloud + monetize); the other spelling of each is UNVERIFIED, so the mock accepts both rather than invent an answer | [`http`] — ledger L112 |
//!
//! Rows 39–63 are lane T13's, from `BEHAVIOURS-LEDGER.md` beside this crate,
//! where every one carries its source. Three rows above are **not UpCloud's**:
//! 3 and 37 (`412 out_of_stock`) and 5 ("0 rows + 403") were verified against
//! Scaleway (fr-par-1 2026-09-09; the IAM change 2026-08-26), not UpCloud.
//! Owner ruling 2026-09-21: they are kept as generic cloud-provider faults —
//! relabelled, not removed, because removing them would re-map every storm
//! seed — and they are never to be read as UpCloud behaviour.
//!
//! Behaviours 35 and 36 exist for ONE caller: `private-gunnar-ops`'s
//! `estate::prune_dead_firewalls` and the row F38 that proves it. Together with
//! 1 and 5 they are the four ways a `403`/`404` pair can lie about whether a
//! server is there, and a caller that reads the list, the detail AND the
//! firewall before it drops a state entry is the only caller that survives all
//! four. A stale state naming a server that was re-laid under a NEW uuid — the
//! likeliest reading of 2026-09-20 — needs no fault at all: create the estate,
//! ask about a uuid that was never created, and the mock answers 404 on the
//! server and 403 on its firewall while the list shows the same titles alive.
//!
//! # Two callers, one API
//!
//! Everything above is the surface `monetize-cloud-impl`'s plugin and
//! `gunnar/deploy/upcloud` drive. `private-holger-ops` drives the SAME account
//! through terraform and `UpCloudLtd/upcloud` 5.44, which carries
//! `UPCLOUD_DEBUG_API_BASE_URL` — so the real declaration and the real provider
//! can be pointed here too, and holger's twin, which has never been created
//! because creating it costs money, can be laid down for nothing. That door is
//! [`tf`], and the first thing it proved was about this crate rather than about
//! holger: a device's `storage_size` was rendered as a STRING here, the
//! official provider's own SDK declares it `int`, and every terraform server
//! create died on a shape the account cannot send. Behaviour 10's lesson, a
//! second time.
//!
//! # The calibration
//!
//! Every fault above is a claim that the mock can be wrong in a particular way,
//! and a claim nobody has checked is a claim. [`self_check`] drives EVERY
//! variant in [`Fault::ALL`] twice — disarmed and armed — and fails if the two
//! observations are the same, because a fault that changes nothing a caller can
//! see advertises a coverage this crate does not have. `mock-upcloud
//! --self-check` prints the table and exits non-zero. Behaviour 38 is what it
//! is for: `Estate::detach` had no path that answered `200` without removing
//! the device, so every sweep driven against this mock passed a test the mock
//! could not fail.
//!
//! # Time
//!
//! Every duration above is real, and running 100 000 of them at real speed is
//! three years. So the mock keeps a **virtual clock** ([`clock::Clock`]) with a
//! `speed` knob: `1.0` is faithful wall time, `0.0` makes every transition
//! instant, `100.0` is a hundred times quicker. The ORDER and the RELATIVE
//! durations never change — a storage still goes `maintenance` → `maintenance` →
//! `online`, a delete still traverses `maintenance`, and a client that does not
//! poll still fails. What changes is only how long the harness waits.
//!
//! This is the reason the mock is a state machine over a clock rather than a
//! table of canned replies: a canned reply cannot be *early*.
//!
//! # Faults
//!
//! [`faults::Faults`] is a set. A fault can be armed by name (deterministic: it
//! fires every time it can) or a **seed** can be handed in, in which case the
//! fault weather for the whole run is derived from that seed alone. A failing
//! run is replayable from its seed and nothing else — no recording, no capture
//! file. See [`rng::SplitMix64`]: the generator is thirty lines and in this
//! crate on purpose, because a dependency's PRNG may change its stream between
//! versions and then a seed stops naming a run.
//!
//! # Behaviour 10 was wrong, and the correction is the more useful entry
//!
//! This crate shipped claiming the API sends no `created` on storage rows, so a
//! young volume and an old orphan were the same row. That was the brief and the
//! brief was mistaken. MEASURED against the live account 2026-09-20 — two
//! volumes, both endpoints, `created` present every time:
//!
//! ```text
//! 018a7b17-5698-4aa9-b160-f6a6daea9921
//! GET /1.3/storage/private → "created": "2026-09-20T10:35:18Z"
//! GET /1.3/storage/{uuid} → "created": "2026-09-20T10:35:18Z"
//! the Resize Backup's detail → "created": "2026-09-19T21:54:06Z"
//! ```
//!
//! **A mock stricter than the provider is worse than one that is laxer.** This
//! crate wrote that sentence itself, about the virtio hot-plug attach it used to
//! refuse and which manufactured 82 % of a storm's failures. The same mistake,
//! twice, in opposite directions: a lax mock misses a bug, a strict one invents
//! a verdict path the account cannot reach. It is written here rather than
//! quietly fixed because the correction is worth more than the entry was.
//!
//! What remains true is smaller and still worth having: the `Resize Backup`
//! carries no product name in its title (behaviour 9), so a TITLE filter is
//! blind to it — and that is what `labelled_gunnar` in `private-gunnar-ops`
//! now exists for. The date was never the missing half; the label was.
//!
//! The storage rows the mock sends carry `created` because that is MEASURED.
//! The server rows carry it too, which is INFERRED from the same API sending it
//! for storages and has not been measured on its own — marked here rather than
//! presented as fact, and erring toward sending it for the reason above.
//!
//! # What it is NOT
//!
//! It is not a simulator of UpCloud's billing, its IP transfer market, its
//! network products (routers, gateways, load balancers, object storage), its
//! managed databases or its Kubernetes. It answers the subset this estate
//! drives, listed in [`http`], and answers `404` with a `NOT_IMPLEMENTED`
//! envelope naming the path for everything else — loudly, so an unimplemented
//! call can never read as a working one.
pub use ;
pub use Estate;
pub use ;
pub use ;
pub use ;