mock-upcloud 0.1.3

A faithful fake of the UpCloud API 1.3 — the lies included — backed by real KVM guests
Documentation

mock-upcloud — the UpCloud API 1.3, faithfully wrong.

A real HTTP server that answers UpCloud's 1.3 surface out of a state machine. Point UPCLOUD_API_BASE at it and the real monetize-cloud-impl upcloud plugin, the real gunnar/deploy/upcloud procedure and the real cargo xtask estate verbs talk to it without knowing. Nothing here reaches the network.

The point is the lies, not the happy path

A mock that answers every call correctly proves nothing, because that is not the provider anybody has. Every behaviour below was MEASURED against the live account or is written down in this estate's history (3, 5 and 37 are the exception: verified against Scaleway, not UpCloud — see below), and each one is armable on its own ([faults::Fault]) so a defect can be reproduced on demand instead of waited for.

# behaviour where it is modelled
1 a firewall-rule read for a DELETED server answers 403 ERROR_AUTHENTICATION_FAILED, not 404 [http::firewall_rule], always on — it is not a fault, it is the API
2 GET /1.3/price fails at the TRANSPORT level, not with a status [Fault::PriceTransportReset] — the socket is closed mid-reply
3 412 out_of_stock on poweron, for days — verified against Scaleway (fr-par-1 2026-09-09), not UpCloud; kept as a generic cloud-provider fault (UpCloud documents 409 STORAGE_RESOURCES_UNAVAILABLE) [Fault::OutOfStock], sticky until disarmed
4 the server LIST carries no storage_devices; the storage list DOES carry labels [render], always on
5 a revoked credential answers 0 servers and 403 on some endpoints, never a clean 401 — verified against Scaleway (IAM change 2026-08-26), not UpCloud; kept as a generic cloud-provider fault (a dead UpCloud token is a clean 401: 43) [Fault::RevokedCredential]
6 create a server: 98–105 s [clock::Timings::server_create]
7 delete front/twin ≈ 60 s; delete an appliance carrying 4 member volumes > 5 min in maintenance [clock::Timings::server_delete]
8 the guest's own install ≈ 10 s (INSTALL-OK 10 011 ms measured under KVM) [clock::Timings::guest_install] / the KVM half
9 stop-resize-start leaves an auto-titled Resize Backup, detached, labelled, WITHOUT "gunnar" in the title [estate::Estate::modify_server]
10 young objects are indistinguishable from orphans without created WRONG, and corrected below created is SENT, on both the list and the detail; [Fault::WithholdCreatedField] takes it away on request
11 a server must be stopped (hard) before it can be deleted with its storages [estate::Estate::delete_server]
12 utility IPs are reassigned between re-lays; public IPs change every time [estate::Addresses], shuffled per lay from the seed
13 no serial console — VNC only; a PID-1 panic is invisible except on the framebuffer the KVM half boots -serial none
14 the VNC port goes stale on every stop/start and the API keeps reporting the old one [Fault::StaleVncPort] (default ON — it is the provider's normal)
15 SeaBIOS, no UEFI the KVM half refuses an OVMF spec by name
16 cdrom/eject works on a STARTED server; IDE/virtio detach does not (the virtio half is the guest's — see 60) [estate::Estate::eject] / [estate::Estate::detach]
17 a re-image killed mid-install loops the installer (CD stays first in the boot order) [estate::Guest::boot]
18 inbound UDP replies are dropped (DNS/NTP) [Fault::UdpInboundDropped], default ON; [guest_clock::ntp_answer] answers silence
19 a guest boots one TIMEZONE OFFSET ahead — +2 h in summer, +1 h in winter — and not every guest does [guest_clock]: the hypervisor's RTC is correct UTC, the GUEST misreads it ([Fault::GuestReadsRtcAsLocalTime])
20 the installer PID 1 brings no network up, so nothing narrates the install window the KVM half's install image
21 on first boot unattended-upgrade-shutdown --wait-for-signal holds dpkg lock-frontend for the life of the boot the Ubuntu template guest
22 a volume NEVER shrinks; a growth is permanent and billed forever [estate::Estate::modify_storage] refuses a shrink by name
23 the direct-upload import and the storage have two different clocks: the import says completed five seconds in, the storage sits in syncing for 100–130 s more [estate::Estate::upload] → [estate::After::BeginSync]
24 the import object carries read_bytes/written_bytes and REAL md5sum/sha256sum — the ladder verifies the sha256 against the local file [digest], which computes them rather than inventing them
25 the same 43 MB medium is uploaded TWICE per re-image (cdrom, then virtio, because korp-installer probes virtio alone) — two imports, two syncs, ~4 min of pure waiting [clock::Timings::storage_sync_lo_ms]
26 POST /storage/{uuid}/clone is the candidate fix, and whether it skips the sync is MEASURED after all — see 53 [estate::Estate::clone_storage]; pessimistic by default, [Fault::CloneSyncsLikeImport] for the optimistic guess
27 one wrong clock is SIX red rows: skew-refusal → no-quorum → clock-unchanged → console-key → banner → monetize-poll [guest_clock::cascade]
28 the skew window is never widened: a row that needs a wider window is a RED [guest_clock::widening_the_window_still_leaves_a_red] — no number turns a wrong clock green
29 DHCP option 121 is offered and a guest that ignores it has NO ROUTE off its /22 — outbound dies, inbound stays healthy [net::outbound_reach] / [Fault::GuestIgnoresDhcpOption121]
30 hairpin NAT does not exist: the box holding a DNAT cannot use it on its own public address [net::Reach::NoHairpin] — and it reads connection refused
31 the console is a HOST and a port, and the toggle re-provisions BOTH (se-sto1.vnc.upcloud.com:60031) [estate::Server::console], [render::server]
32 the guest's install is 2–5 s on the UART while the ladder's wall measurement of the same re-image is 110–253 s [clock::Timings::reimage_observers] — two observers, two orders of magnitude
33 a re-image mints a NEW SSH host key every time; three paths must agree to tell that from a hijacked name [estate::Estate::host_key_via] / [Fault::HijackedName]
34 the door that actually leaks: POST /storage/{uuid}/resize (the twin's xfs growth) takes a Resize Backup FIRST, the size of the volume, and hands it back as resize_backup; nothing deletes it. Labelled by default (measured); [Fault::ResizeBackupUnlabelled] is the brief's hypothesis that it carries nothing at all [estate::Estate::resize_filesystem]
35 the firewall of a server that READS FINE answers the same 403 ERROR_AUTHENTICATION_FAILED as behaviour 1 — a credential scoped without the firewall permission; only the server's own detail tells "forbidden" from "gone" [Fault::FirewallForbidden], by name only (INFERRED, not measured)
36 GET /1.3/server/{uuid} answers 404 SERVER_NOT_FOUND for a uuid the LIST carries — the account contradicting itself [Fault::DetailNotFoundForListedServer], by name only (REPORTED 2026-09-20, uuids never compared before the box was wiped)
37 a sold-out zone refuses the create with the same 412 out_of_stock it refuses the poweron with, and mints nothing — verified against Scaleway (fr-par-1 2026-09-09), not UpCloud; kept as a generic cloud-provider fault [estate::Estate::create_server] / [Fault::OutOfStock], one sticky fault for both doors
38 a detach answers 200 and the volume stays attached — the write that reports its own success and did not happen; only a read-back tells it from a real one [estate::Estate::detach] / [Fault::DetachSaysSuccessButStaysAttached], by name only (REPORTED)
39 an empty path segment (/1.3//storage) is a 404 that names no path [http], always on — gunnar api.rs, MEASURED
40 a READ answers 502 now and then; a poll must survive it [Fault::ReadBadGateway], by name (REPORTED, gunnar wait.rs)
41 PUT /1.3/server/{uuid} answers 202 [http], always on — MEASURED 2026-09-19
42 a chunked PUT to the direct-upload URL answers 411 [http], always on (REPORTED, gunnar http.rs)
43 a DEAD token is a clean 401 AUTHENTICATION_FAILED on every call [Fault::DeadToken] — MEASURED 2026-09-14 (ROTATION §2.3)
44 past the account's MaxIOPS quota (yvra: 10 240 GiB) a create or grow is 403 MAXIOPS_STORAGE_LIMIT_REACHED [estate::Estate::create_storage] / modify_storage, always on (limit MEASURED, code REPORTED)
45 a disk boot reads started ~10 s after the start (was 20 s here) [clock::Timings::start_ms] — MEASURED 2026-09-19
46 an installer that powers the box off never reads started: maintenance t+10…t+130 s, then stopped [clock::Timings::install_pass_lo_ms], [estate::After::GuestPoweredOff] — MEASURED 2026-09-15/19
47 stop_server with timeout_action is a 400 (it belongs to restart) [http], always on — MEASURED 2026-09-14
48 a plan change is done when the PUT answers (the provider starts at once); a start during maintenance is 409 [estate::Estate::modify_server] / start_server — the provider's order MEASURED 2026-09-21
49 a boot_order PUT does not end an install loop; it is read at the next HYPERVISOR start always on, now tested — MEASURED 2026-09-14
50 the VNC toggle settles ~2 s after the "no"; a "yes" inside the window gives the OLD endpoint back [clock::Timings::vnc_settle_ms] — MEASURED (gunnar console.rs)
51 the console toggle is 409 while the server is maintenance [estate::Estate::modify_server] — MEASURED (memory: upcloud-vnc-port-toggle)
52 an import's storage ends syncingmaintenanceonline [clock::Timings::sync_tail_maintenance_ms] — sequence MEASURED, split not
53 a clone is maintenanceonline in 47 s with NO syncing (26's "not measured" was measured) [clock::Timings::clone_online_ms]; the old guess is [Fault::CloneSyncsLikeImport] — MEASURED 2026-09-20
54 a shrink AND a same-size PUT are 400 SIZE_INVALID "The new size must be greater than the old size." [estate::Estate::modify_storage] — MEASURED 2026-09-14 (22's code was wrong)
55 a size grow is maintenance for 37 s, then online [clock::Timings::storage_grow_ms] — MEASURED 2026-09-14
56 a grow of a storage attached to a RUNNING server is 409 STORAGE_ATTACHED (terraform 5.44.1 sends it without stopping) always on; [Fault::ResizeRequiresDetach] for the stricter docs page — REPORTED (docs)
57 a storage already attached is 409 STORAGE_ATTACHED on a second attach [estate::Estate::attach_at] — REPORTED
58 a requested address is honoured, virtio takes the first FREE slot, and the guest names virtio disks by slot order, contiguously (vda = virtio:0) [estate::Estate::attach_at], [estate::Estate::guest_disk_names], /mock/disks/{uuid}
59 a storage in maintenance/syncing, or the source of a clone still being made, cannot be deleted: 409 STORAGE_STATE_ILLEGAL [estate::Estate::delete_storage] — REPORTED
60 virtio hot-plug is the GUEST's: a hot-plug kernel (needs PCI_MSI) takes an attach/detach on a running server; one without answers 511 HOTPLUG_FAILED (16's 409 VIRTIO_HOTPLUG_UNSUPPORTED was invented) [Fault::GuestKernelLacksHotplug], per server at create — MEASURED 2026-09-14
61 an interface that asks for IPv6 gets IPv6 ("no IPv6" is the estate's law, not UpCloud's) [render::server] — REPORTED
62 the firewall is evaluated: off or no rules = open; top-down first match; a drop is silence; the utility net is filtered; UDP replies die behind a catch-all drop [net::firewall_admits], /mock/inbound/{uuid}, /mock/udp-reply/{uuid} — MEASURED 2026-09-13/14/20
63 the mock says whom it heard (per credential digest), so a terraform verb can prove the provider talked to THIS mock before apply/destroy [http::Mock::heard_from], /mock/heard?token_sha256=
64 a server PUT changes hostname/title in place (the mock dropped both; terraform refused its own apply) [estate::Estate::rename_server] — MEASURED 2026-09-21 through 5.44.1
65 POST /server/{uuid}/cdrom/load puts a medium into an EMPTY tray; a loaded one is 409 CDROM_DEVICE_IN_USE [http] — L110, DOCUMENTED, not measured
66 an import straight after POST /storage (no wait) is accepted; any other non-online state is still refused [estate::Estate::start_import], [estate::After::Created] — acceptance MEASURED (live receipt 2026-09-21, gunnar plan.rs:550); the state it read then is not recorded
67 the direct-upload PUT answers the import object BARE ({"written_bytes": …, "sha256sum": …}), not wrapped in storage_import [http] — MEASURED: the live re-image of 2026-09-21 08:42 read it at the top level, done: true
68 the verified spellings: a storage size is sent as a JSON NUMBER (verified live, monetize grow 2026-09-14), a stop timeout as a STRING (verified live, gunnar-upcloud + monetize); the other spelling of each is UNVERIFIED, so the mock accepts both rather than invent an answer [http] — ledger L112

Rows 39–63 are lane T13's, from BEHAVIOURS-LEDGER.md beside this crate, where every one carries its source. Three rows above are not UpCloud's: 3 and 37 (412 out_of_stock) and 5 ("0 rows + 403") were verified against Scaleway (fr-par-1 2026-09-09; the IAM change 2026-08-26), not UpCloud. Owner ruling 2026-09-21: they are kept as generic cloud-provider faults — relabelled, not removed, because removing them would re-map every storm seed — and they are never to be read as UpCloud behaviour.

Behaviours 35 and 36 exist for ONE caller: private-gunnar-ops's estate::prune_dead_firewalls and the row F38 that proves it. Together with 1 and 5 they are the four ways a 403/404 pair can lie about whether a server is there, and a caller that reads the list, the detail AND the firewall before it drops a state entry is the only caller that survives all four. A stale state naming a server that was re-laid under a NEW uuid — the likeliest reading of 2026-09-20 — needs no fault at all: create the estate, ask about a uuid that was never created, and the mock answers 404 on the server and 403 on its firewall while the list shows the same titles alive.

Two callers, one API

Everything above is the surface monetize-cloud-impl's plugin and gunnar/deploy/upcloud drive. private-holger-ops drives the SAME account through terraform and UpCloudLtd/upcloud 5.44, which carries UPCLOUD_DEBUG_API_BASE_URL — so the real declaration and the real provider can be pointed here too, and holger's twin, which has never been created because creating it costs money, can be laid down for nothing. That door is [tf], and the first thing it proved was about this crate rather than about holger: a device's storage_size was rendered as a STRING here, the official provider's own SDK declares it int, and every terraform server create died on a shape the account cannot send. Behaviour 10's lesson, a second time.

The calibration

Every fault above is a claim that the mock can be wrong in a particular way, and a claim nobody has checked is a claim. [self_check] drives EVERY variant in [Fault::ALL] twice — disarmed and armed — and fails if the two observations are the same, because a fault that changes nothing a caller can see advertises a coverage this crate does not have. mock-upcloud --self-check prints the table and exits non-zero. Behaviour 38 is what it is for: Estate::detach had no path that answered 200 without removing the device, so every sweep driven against this mock passed a test the mock could not fail.

Time

Every duration above is real, and running 100 000 of them at real speed is three years. So the mock keeps a virtual clock ([clock::Clock]) with a speed knob: 1.0 is faithful wall time, 0.0 makes every transition instant, 100.0 is a hundred times quicker. The ORDER and the RELATIVE durations never change — a storage still goes maintenancemaintenanceonline, a delete still traverses maintenance, and a client that does not poll still fails. What changes is only how long the harness waits.

This is the reason the mock is a state machine over a clock rather than a table of canned replies: a canned reply cannot be early.

Faults

[faults::Faults] is a set. A fault can be armed by name (deterministic: it fires every time it can) or a seed can be handed in, in which case the fault weather for the whole run is derived from that seed alone. A failing run is replayable from its seed and nothing else — no recording, no capture file. See [rng::SplitMix64]: the generator is thirty lines and in this crate on purpose, because a dependency's PRNG may change its stream between versions and then a seed stops naming a run.

Behaviour 10 was wrong, and the correction is the more useful entry

This crate shipped claiming the API sends no created on storage rows, so a young volume and an old orphan were the same row. That was the brief and the brief was mistaken. MEASURED against the live account 2026-09-20 — two volumes, both endpoints, created present every time:

018a7b17-5698-4aa9-b160-f6a6daea9921
  GET /1.3/storage/private  → "created": "2026-09-20T10:35:18Z"
  GET /1.3/storage/{uuid}   → "created": "2026-09-20T10:35:18Z"
the Resize Backup's detail  → "created": "2026-09-19T21:54:06Z"

A mock stricter than the provider is worse than one that is laxer. This crate wrote that sentence itself, about the virtio hot-plug attach it used to refuse and which manufactured 82 % of a storm's failures. The same mistake, twice, in opposite directions: a lax mock misses a bug, a strict one invents a verdict path the account cannot reach. It is written here rather than quietly fixed because the correction is worth more than the entry was.

What remains true is smaller and still worth having: the Resize Backup carries no product name in its title (behaviour 9), so a TITLE filter is blind to it — and that is what labelled_gunnar in private-gunnar-ops now exists for. The date was never the missing half; the label was.

The storage rows the mock sends carry created because that is MEASURED. The server rows carry it too, which is INFERRED from the same API sending it for storages and has not been measured on its own — marked here rather than presented as fact, and erring toward sending it for the reason above.

What it is NOT

It is not a simulator of UpCloud's billing, its IP transfer market, its network products (routers, gateways, load balancers, object storage), its managed databases or its Kubernetes. It answers the subset this estate drives, listed in [http], and answers 404 with a NOT_IMPLEMENTED envelope naming the path for everything else — loudly, so an unimplemented call can never read as a working one.