# UpCloud behaviours ledger: everything the estate has learned, reconciled against `mock-upcloud`
Lane T13, 2026-09-21. The question this file answers is the owner's: *"have you
collected all behaviours and misunderstandings using UpCloud, so you can mimic
timing and behaviour?"*
**Sources swept.** Every memory note that mentions UpCloud (22). The code and
docs in gunnar (`master`), private-gunnar-ops (`main`), private-holger-ops
(`main`), holger (`main`), tunnr (`main`), monetize-impl (`main`), monetize
(`main`) and edda (`agent/r41-warehouse-resolver`). This lane's own runs of the
real provider `UpCloudLtd/upcloud` 5.44.1 against the mock (terraform 1.16.2).
All provider runs used a fake token, a loopback base URL, and scratch copies of
both estates' `terraform/`.
**Column key.**
- **M/R.** M = measured against the live account, or with the real provider
binary. R = reported: docs, fixtures or inference. K = measured under KVM,
not on UpCloud.
- **mock.** `no` = not in the mock at all. `desc` = described only. `always` =
always-on. `fault` = an armable `Fault`. `wrong` = the mock encodes something
different. `n/a` = not the provider's behaviour, so the mock must NOT carry it.
- **#.** The behaviour number in `src/lib.rs`'s table. `+NN` marks a number
added by this lane (see the table in `lib.rs`).
Short repo prefixes: **G** gunnar · **PGO** private-gunnar-ops · **PHO**
private-holger-ops · **MI** monetize-impl · **ED** edda · **T** tunnr · **H**
holger · **mem:** a memory note.
---
## 1. API surface and wire shapes
| # | What UpCloud does | Source | M/R | Timing | mock | # |
|---|---|---|---|---|---|---|
| L1 | Base is `https://api.upcloud.com/1.3`, with the version in the path. A doubled slash (`/1.3//storage`) answers 404 with a message that names no path | G deploy/upcloud/src/api.rs:22-24,602-605 | M | – | **no**: the mock collapses empty segments and answers | +39 |
| L2 | The terraform provider appends `/1.3` itself. `UPCLOUD_DEBUG_API_BASE_URL` must NOT end in `/1.3`, or the provider calls `/1.3/1.3/account` | T3 run 2026-09-21 (scratch/t3-mock-estate/mock.log) | M | – | always: the mock answers 404 NOT_IMPLEMENTED on `/1.3/1.3/…`. The contract test pins it | contract |
| L3 | A bare `GET /storage` lists every public template, thousands of rows. `/storage/private` is the one a cleanup uses | G api.rs:62-66 | R | – | always | – |
| L4 | The server LIST carries no `storage_devices`. The storage list does carry `labels` | MI U/api.rs:121,244 | R | – | always | 4 |
| L5 | Numbers arrive as a number in one reply and a string in the next (`size`, `storage_size`, `core_number`, `memory_amount`, `boot_disk "0"/"1"`, `remote_access_port`, `credits`) | G api.rs:11-13; MI U/api.rs:6-8,276 | M/R | – | always (`the_inconsistent_stringification_is_reproduced`) | – |
| L6 | A device's `storage_size` is an INTEGER in the provider's SDK. A string kills every terraform create | T3 2026-09-21 | M (provider) | – | always (fixed on mock-estate-gunnar) | tf |
| L7 | Some write bodies want strings. A numeric stop `timeout` was accepted and silently ignored | PGO grow_cloud.rs:73-76,207-208; PGO upcloud.rs:553-555 | M | – | no: the mock ignores `timeout` either way. Harmless | – |
| L8 | Errors are `{"error":{"error_code","error_message"}}`. A 403 auth failure is problem+json with `correlation_id` and no `error` object | G api.rs:544-565; PGO upcloud.rs:100-112 | M | – | always | 1 |
| L9 | The API occasionally answers **502** on a read, so a failed poll is not fatal | G wait.rs:102-106 | R | – | **no**: the mock only has 503-on-write | +40 |
| L10 | Labels come in two shapes: storage `[{key,value}]`, server `{"label":[…]}`. Filtered with `?label=k%3Dv`. Keys are 2–32 printable ASCII and cannot start with `_` | MI U/api.rs:9-10; MO monetize-cloud/src/lib.rs:44 | R | – | always for the shapes; **no** for key validation (the error code is unknown, so it is not modelled) | – |
| L11 | `created` is sent on storage rows, on both the list and the detail | PGO upcloud.rs:682-685 | M 2026-09-20 | – | always, plus `WithholdCreatedField` (a hypothesis, rate 0) | 10 |
| L12 | Server create answers **202** in the docs and fixtures. The mock answers 201. The provider accepts both | MI U/tests.rs:655 | R | – | desc (left at 201: unmeasured) | – |
| L13 | `PUT /server/{uuid}` (boot_order) answers **202** | PGO RESUME-2026-09-19:27-32 | M | – | **wrong**: mock 200 | +41 |
| L14 | A chunked PUT to the direct-upload URL answers **411 Length Required** | G http.rs:236-242 | R | – | **wrong**: mock 400 MOCK_UPCLOUD_CHUNKED_REQUEST | +42 |
| L15 | The direct-upload URL is the credential. No bearer token is sent to it | G api.rs:175-177; G http.rs:67-70 | R | – | always | – |
## 2. Credentials and accounts
| # | What UpCloud does | Source | M/R | Timing | mock | # |
|---|---|---|---|---|---|---|
| L16 | A **dead** UpCloud token answers `401 AUTHENTICATION_FAILED`, a clean 401 | PGO ROTATION-2026-09-14.md:126-130 | M | – | **no**: the mock only has "no header = 401" | +43 |
| L17 | A live token on an overdrawn account (credits −963.207) still answers 200 | PGO ROTATION:130 | M | – | always (nothing checks credits) | – |
| L18 | A "revoked credential" answers lists 200 with ZERO rows and 403 on details. **Verified against Scaleway, not UpCloud; kept as a generic cloud-provider fault** (owner ruling 2026-09-21): the only dated measurement (2026-08-26) is the Scaleway IAM change (mem: scaleway-iam-revoked-2026-08-26: "0 servers, 403 on VPC"). A dead UpCloud token is a clean 401 (behaviour 43, fault `DeadToken`) | PGO upcloud.rs:28-37; MI U/mod.rs:1055-1077 | R (for UpCloud) | – | fault `RevokedCredential` | 5 |
| L19 | A token for another account gets `403 SERVER_FORBIDDEN` on a server uuid of the first account. A token for the wrong estate authenticates and succeeds everywhere | PGO iso.rs:778-790; G deploy/upcloud/src/token.rs:6-32 | M | – | no: the mock has one account | – |
| L20 | Tokens start `ucat_`, are 31 chars, Bearer. They are account-wide (only `expires` and `allowed_ip_ranges`). Sub-accounts can be scoped; whether a sub-account token inherits the scope is unmeasured | G token.rs:69-70; PGO DATA-SET-GROWTH-FLOW.md:63,345 | R | – | desc | 35 |
| L21 | A `GET /1.3/price` transport failure surfaced as "credential could not be verified" | PGO remote/net.rs:8-15; MI storm/src/main.rs:428 | M | – | fault `PriceTransportReset` | 2 |
| L22 | A firewall-rule read for a DELETED server answers 403 `ERROR_AUTHENTICATION_FAILED`, not 404. terraform refresh dies "Permission denied (status=403)" | PGO estate.rs:925-944 | M 2026-09-20 | – | always | 1 |
| L23 | The list carries a uuid that the detail answers 404 for | PGO estate.rs:1227-1231 | R | – | fault `DetailNotFoundForListedServer` | 36 |
| L24 | A live server's firewall answers the same 403 (a permission-scoped credential) | inferred | R | – | fault `FirewallForbidden` | 35 |
| L25 | Quota exhaustion is `403 *_LIMIT_REACHED` (e.g. `MAXIOPS_STORAGE_LIMIT_REACHED`). The yvra `storage_maxiops` limit is 10 240 GiB | PGO DATA-SET-GROWTH-FLOW.md:212; DESIGN:31 | M (limit) / R (code) | – | **no** | +44 |
| L26 | `402 INSUFFICIENT_CREDITS` exists. UpCloud suspends resources when prepaid credits run out | MI U/api.rs:552-560; PGO gunnar_rs_system.md:970 | R | – | no (billing is out of scope) | – |
## 3. Server state machine and timing
| # | What UpCloud does | Source | M/R | Timing | mock | # |
|---|---|---|---|---|---|---|
| L27 | Server states are `started · stopped · maintenance · error` | G api.rs:271; MI U/api.rs:239 | R | – | always (no `error`) | – |
| L28 | Create server: `maintenance` for 98–105 s | lib.rs history | M | 98–105 s | always | 6 |
| L29 | Delete: front/twin ≈60 s; an appliance with 4 member volumes stays in `maintenance` >5 min | lib.rs history | M | 60 s / >300 s | always | 7 |
| L30 | **terraform destroy reports complete while the 4-member appliance is still `maintenance`**. The provider does not wait for the server to vanish; it deletes the template disk and the data volumes right after the DELETE | T13 run: DELETE server → DELETE storage within the same second, servers still `maintenance` at "Destroy complete" | M (provider) | – | always (reproduced by construction, see §9) | 7 |
| L31 | Delete requires `stopped`. A destroy is verified by GET until 404, not by the DELETE status | PGO ROTATION:135-137; G gunnar-loadbench/src/upcloud.rs:22-43 | R | – | always | 11 |
| L32 | Start → `started` in **~10 s** for a disk boot | PGO RESUME-2026-09-19:32 | M | 10 s | **wrong**: `start_ms` = 20 000 | +45 |
| L33 | **A guest that powers itself off inside the start operation reads `started` BRIEFLY, then `maintenance`, then `stopped`**: live re-image 2026-09-21 08:42, `start` 377 421 ms → `started` read 381 091 ms → `maintenance` 381 174 ms → `stopped` 530 945 ms. The 2026-09-19 reading "`maintenance` t+10…t+130 s → `stopped` t+140 s, never `started`" was polled every 10 s from t+10 s and could not see a window this short; its maintenance/stopped half stands. A waiter that wants `started` alone misses the window at a 2 s poll and times out on `stopped` (the 2026-09-19 18:12 run); `--medium-powers-off` (G plan.rs:676-714) accepts either | PGO .reimage/reinstall-receipt.json (2026-09-21); PGO RESUME-2026-09-19:13-25; G plan.rs:668-714 | M | `started` at ~3.6 s for < 100 ms; pass 130–154 s | always: `started` 3.6–3.7 s, `maintenance` to the pass end, `stopped` | 46 |
| L34 | A soft stop on a wedged guest hangs for the full timeout, then the stop fails. The tools fall back to hard | G main.rs:461; PGO grow_cloud.rs:210-245 | R | 60/120 s | always (Looping/Panicked refuse a soft stop) | – |
| L35 | `stop_server` has no `timeout_action` → **400**. That field belongs to restart | mem: upcloud-reimage-install-loop | M | – | **no** | +47 |
| L36 | Stop/start on a server already in that state → `409 SERVER_STATE_ILLEGAL` | MI U/mod.rs:743,751 | R | – | always | – |
| L37 | "Stopped" did not mean off: both hentor boxes still answered the next day | PGO gunnar_rs_system.md:920-929 | M | – | desc (T12's `reap` keeps the API from saying `stopped` over a running VM) | timing |
| L38 | The API never says an install finished. A box that answers nothing (xfs duplicate-uuid, no ICMP/22/2222) still reads `started` | G plan.rs:46-49; mem: gunnar-iso-only-as-fixed-as-its-pin | M | – | always (state never reads guest health; `Guest::Panicked`) | 38-ish, see +46 |
| L39 | Plan change on a started server → `409 SERVER_STATE_ILLEGAL`. Only a stopped server may change plan | MI U/mod.rs:73-80 | R | – | always | – |
| L40 | A plan change mints a `Resize Backup`. **Unconfirmed:** the only measured 44 GB Resize Backup came from the twin's FILESYSTEM resize (`POST /storage/{uuid}/resize`, L66), not from a plan change | PGO live.rs:1377; lib.rs #9 | R | – | always, and **lost**: the pending mint was dropped when a `start` arrived during the plan-change `maintenance` (found by the provider run, §9) | 9, +48 |
| L41 | Out of stock: `412 out_of_stock` on poweron and on create, for days. **Verified against Scaleway, not UpCloud; kept as a generic cloud-provider fault** (owner ruling 2026-09-21): mem: scaleway-out-of-stock-2026-09-09 records it on Scaleway fr-par-1. UpCloud's documented capacity refusal is `409 STORAGE_RESOURCES_UNAVAILABLE` | PGO grow_cloud.rs:499-503; MI U/api.rs:552-560 | R (for UpCloud) | days | fault `OutOfStock` | 3, 37 |
| L42 | A create that commits and then loses its reply | – | R | – | fault `CommitThenDropReply` | – |
| L43 | Two storages are provisioned **in parallel** by UpCloud; only a serial client made it look slow | G lib.rs:79-84 | M | 2×110 s → ~115 s | always (independent transitions) | – |
## 4. Boot, firmware, console
| # | What UpCloud does | Source | M/R | Timing | mock | # |
|---|---|---|---|---|---|---|
| L44 | SeaBIOS only, no UEFI, no firmware switch. A UEFI-only medium gives "noll utdata"; SeaBIOS executes the FAT `efiboot.img` | ED iso_writer.rs:452-460; G installtest/firmware.rs:70-72 | M | – | always (KVM half refuses OVMF) | 15 |
| L45 | **The installed disk must boot under SeaBIOS too**: a UEFI-only installed disk never boots. The fix is an MBR plus a BIOS stage. KVM proves it boots under SeaBIOS/i440fx | ED KI/src/install.rs:879-885; mem: tunnr-kernel-hotplug | M | – | desc: the virtual guest assumes any boot disk boots; T12's `DraupnirGuest` boots the real disk under SeaBIOS | 15 (T12) |
| L46 | Machine shape: `-machine pc` (i440fx), 2 virtio NICs (public, utility), virtio-blk boot disk, CPU `AMD EPYC 7542` (EPYC-Rome), ACPI, metadata at 169.254.169.254. The bus topology is unmeasured | ED BT/target/presets.rs:1050-1071; G probe.rs:99-102 | M | – | desc (T12's guest) | – |
| L47 | **No UART at ttyS0/0x3f8**; ttyS1/ttyS2 exist at 0x2f8/0x3e8. With `console=ttyS0`, every write gets EIO and PID 1's first `print!` panics. VGA over VNC is the only console | ED KI/src/serial.rs:258-268; G firmware.rs:72-76 | M 2026-08-29 | – | desc: "no serial console" (the KVM half boots `-serial none`); ttyS1/2 are not modelled | 13 |
| L48 | No QMP, no serial console API. VNC is the only screen | PGO screendump.rs:37-41 | R | – | always | 13 |
| L49 | The ISO must be attached TWICE: an IDE cdrom for SeaBIOS and a virtio disk for Linux (the kernel has no ATA). SeaBIOS cannot boot the virtio copy | G deploy/upcloud/src/lib.rs:8-32 | M | – | always | 25 |
| L50 | `boot_order` can be written on a started server but is honoured only at the next HYPERVISOR start; a guest reboot keeps the firmware order | G api.rs:273-274; mem: upcloud-reimage-install-loop | M | – | always (implicit: a looping guest stays looping after a PUT). Now tested | 17, +49 |
| L51 | A re-image killed mid-install loops install→reboot (CD first) for 2+ h, rewriting the disk each pass | G plan.rs:10-17 | M | 2 h+, each pass ≥900 s | fault `InstallerLoop` | 17 |
| L52 | `POST /server/{uuid}/cdrom/eject` works on a STARTED server (200); the device stays with an empty `storage`; the next reboot boots the disk | G api.rs:101-108 | M | eject at 120 s is safe | always | 16 |
| L53 | Ejecting too early (before SeaBIOS has read kernel and initrd) falls back to the old disk | G plan.rs:50-58 | M | <120 s | no (the virtual guest has no early window) | – |
| L54 | Accepted boot orders: `disk`, `cdrom,disk`, `disk,cdrom`, `cdrom`. `cdrom,disk` with no medium falls through to the disk | PGO estate.rs:430-431 | R | – | always (`disk,cdrom` reads back as `disk`) | – |
| L55 | Guests boot +2 h (CEST) or +1 h (CET): no kvmclock (`CONFIG_HYPERVISOR_GUEST` off) → reads CMOS as local time. The Ubuntu front is within 468 ms | G recipe.rs:584-610 | M (mechanism inferred) | 7 198 668 ms | fault `GuestReadsRtcAsLocalTime` | 19 |
| L56 | A re-image mints a NEW SSH host key every time (also `[host]:2222`). A re-created front keeps its public IPv4, so the key changes on the same address | mem: gunnar-estate-3vm-2026-09-15; PGO front.rs:18-21 | M | – | always, plus `HijackedName` | 33 |
## 5. Console (VNC)
| # | What UpCloud does | Source | M/R | Timing | mock | # |
|---|---|---|---|---|---|---|
| L57 | Plain RFB, no TLS, at `<zone>.vnc.upcloud.com:<port>`. The password is in `remote_access_password`. Banner `RFB 003.008` (read 12 bytes, not 40) | mem: upcloud-vnc-port-toggle | M | – | always (host and port). The mock never reports a real host: `<zone>.vnc.mock.invalid` (RFC 2606) with no guest; with a real guest (`mock-upcloud-kvm`) the guest's own VNC on `127.0.0.1:<port>`, which answers `RFB 003.008`, goes stale on stop/start (14) and is reconciled by the toggle (59) | 31 |
| L58 | The port is reallocated on every stop/start and the API keeps reporting the stale one (a SYN hangs) | mem: upcloud-vnc-port-toggle | M | – | fault `StaleVncPort`, default ON | 14 |
| L59 | Cure: PUT `remote_access_enabled` no → pause → yes, then re-read. The PUT returns before the console moves; **without the ~2 s pause the old port comes back** | G console.rs:17-29,96-99; PGO screendump.rs:896-927 | M | settle 2 s | **no**: the toggle is instant in the mock | +50 |
| L60 | The toggle answers **409 while the server is `maintenance`** (retry) | mem: upcloud-vnc-port-toggle | M | – | **no**: a PUT is accepted in any state | +51 |
| L61 | The toggle rotates the VNC password. The password is 8–32 alphanumerics; VNC-Auth type 2 only | PGO screendump.rs:455-460; G console.rs:67-73 | R | – | no (password is echoed back unchanged) | – |
## 6. Storage: create, import, clone, attach, delete
| # | What UpCloud does | Source | M/R | Timing | mock | # |
|---|---|---|---|---|---|---|
| L62 | Direct-upload import: the PUT takes ~5 s for 43 MB; the import says `completed` while the storage is still `syncing` 100–130 s more (two clocks). Storage path: **maintenance → syncing → maintenance → online** | G lib.rs:55-77; G tests/clone_probe.rs:19-27 | M | PUT 1–5 s; import→online 110–114 s | always; **short one `maintenance`**: the mock goes syncing → online | 23, +52 |
| L63 | The import object carries `read_bytes`/`written_bytes` and real `md5sum`/`sha256sum` (possibly upper-case hex). The uploader's PUT reply is the same fields **un-enveloped**, `{"written_bytes": …, "md5sum": …, "sha256sum": …}`; only `GET /storage/{uuid}/import` wraps them in `storage_import` | G upload.rs:5-8; G plan.rs:472-492 (`resp.json()` read top-level, live `done: true` 2026-09-21) | M | – | always (real digests; the PUT reply was wrongly enveloped until 2026-09-21) | 24 |
| L64 | A truncated upload still boots (ISO-9660 header is at the front) and fails ~15 min later | G upload.rs:11-16 | M | ~15 min | no (a guest property; out of scope) | – |
| L65 | **A clone goes maintenance → online in 47 s with NO `syncing`**; the call takes 728 ms; the tier is inherited. It cannot start until its source is `online` | G tests/clone_probe.rs:5-32 (MEASURED 2026-09-20) | M | 728 ms; 47 s | **wrong**: the mock's default makes a clone sync 100–130 s ("NOT MEASURED"). The measurement exists | 26, +53 |
| L66 | `POST /storage/{uuid}/resize` (filesystem) first takes a `Resize Backup <n>` the size of the volume, hands it back as `resize_backup`, and leaves it billed. It carries the origin's labels and `origin` | PGO upcloud.rs:20-26,538-545; PGO live.rs:1377-1398 | M 2026-09-20 | 44 GB since 2026-09-19 | always | 34 |
| L67 | terraform `filesystem_autoresize` plus `delete_autoresize_backup = true` deletes that backup itself (PUT size → POST resize → DELETE backup) | T13 provider run §9 | M (provider) | – | always (the provider does it; the mock answers) | contract |
| L68 | A storage only grows: smaller → **`400 SIZE_INVALID`** "The new size must be greater than the old size." The same-size retry gets the same error | PGO DATA-SET-GROWTH-DESIGN.md:23; FLOW:271 | M | – | **wrong code**: the mock says `STORAGE_INVALID_SIZE` and accepts a same-size PUT | 22, +54 |
| L69 | A detached 1→2 GiB grow: 200, then `online` again 37 s later | PGO DESIGN:24; mem: gunnar-grow-no-restart | M | 37 s | **no**: the mock grows instantly, with no `maintenance` | +55 |
| L70 | **Resize while attached.** The docs contradict themselves ("server must be `stopped`" vs `STORAGE_ATTACHED` "must first be detached"). Both readings refuse it while the server RUNS. mem: gunnar-grow-no-restart says "attached resize refused" | PGO DESIGN:25,282 | R (docs) | – | **no**: the mock (and so terraform through it) grew four volumes attached to a STARTED appliance, §9 | +56 |
| L71 | Volume ceiling 4096 GB, floor 10 GB (per the estate's code) | PGO grow.rs:919-920 | R | – | no (the codes are unknown) | – |
| L72 | A storage not `online` cannot be attached | G plan.rs:527-543 | R | – | always | – |
| L73 | Attaching a storage that is already attached → `409 STORAGE_ATTACHED`. UpCloud refuses a storage another server holds | MI U/mod.rs:651; PHO front.rs:151-154 | R | – | **no**: the mock attaches the same storage twice | +57 |
| L74 | `address: "virtio"` lets UpCloud pick the slot; a named address (`virtio:3`) is honoured | MI U/grow.rs:70-71 | R | – | **no**: the mock ignores the requested address and takes `virtio:<count>`, which collides after a middle detach | +58 |
| L75 | Delete an attached storage → refused (409). Delete a clone's source while the clone is still being made → refused. A second DELETE → 404 | G api.rs:518-520; PGO upcloud.rs:286-288 | R | – | attached: always; **clone source: no** | +59 |
| L76 | `DELETE /storage/{uuid}` on a storage in the wrong state → `409 STORAGE_STATE_ILLEGAL` | MI U/tests.rs:1032 | R | – | **no**: a storage in `maintenance` deletes | +59 |
| L77 | `DELETE /server/{uuid}?storages=1&backups=keep` takes the OS disk with it | MI U/mod.rs:1022 | R | – | always (`backups` ignored) | – |
| L78 | Detaching does not stop billing; a stopped server still bills its storage | G api.rs:510-516 | M | – | n/a (billing out of scope) | – |
| L79 | A young storage looks like an orphan until `created` is read | PGO upcloud.rs:17-19 | R | – | always | 10 |
## 7. Hot-plug and growth
| # | What UpCloud does | Source | M/R | Timing | mock | # |
|---|---|---|---|---|---|---|
| L80 | Detach from a STARTED server: IDE → `409 IDE_HOTPLUG_UNSUPPORTED`; **virtio → `511 HOTPLUG_FAILED`** (the guest did not ack the ACPI `_EJ0`, i.e. the kernel had no PCI hot-plug) | mem: upcloud-reimage-install-loop; G api.rs:101-106 | M 2026-09-14 | – | **wrong**: the mock answers `409 VIRTIO_HOTPLUG_UNSUPPORTED`, an invented code, and refuses every virtio detach on a started server, even for a guest that can hot-plug | 16, +60 |
| L81 | **Storage hot-plug is a GUEST property.** It needs `HOTPLUG_PCI(_ACPI/_PCIE/_SHPC)`, `PCIEPORTBUS` and **`PCI_MSI=y`** (without MSI, `_OSC` refuses OS control and pciehp is dead). Proven only under KVM; never measured on UpCloud | T tunnr-hardening.config:170-188; mem: tunnr-kernel-hotplug | K | eject ack ~44 ms | **no**: the mock has no notion of the guest's kernel | +60 |
| L82 | A virtio ATTACH to a started server answers `HOTPLUG_FAILED` with the old kernel. One source says 409 SERVER_STATE_ILLEGAL "The server is started." for attach-to-started; HOLGER-PLAN says attach works started or stopped. Hot-attach is **unmeasured** on UpCloud | mem: tunnr-kernel-hotplug; G api.rs:332; H HOLGER-PLAN.md:1726-1732 | M (old kernel) / R | – | **lax**: the mock always accepts a virtio attach on a started server | +60 |
| L83 | Resizes are batched in a 10 s window; growth at 80 %/65 %/90 %, at most once per 3600 s | mem: gunnar-grow-no-restart; MO grow/batch.rs:4-18 | – | 10 s | **n/a: this is the ESTATE's policy, not UpCloud's.** The mock must not batch | – |
| L84 | Members already resized before an interrupted growth stay bigger and stay billed | PGO live.rs:1463-1467 | R | – | always (never shrinks) | 22 |
| L85 | No live growth has ever run on UpCloud (`allow_grow=false`) | PGO STINK-LIST:138-142 | R | – | – | – |
## 8. Network and firewall
| # | What UpCloud does | Source | M/R | Timing | mock | # |
|---|---|---|---|---|---|---|
| L86 | Two NICs: index 1 public (eth0), index 2 utility (eth1). The guest does not bring up the utility NIC by itself | G recipe.rs:1306-1308 | M | – | always (default pair) | – |
| L87 | The utility network is NOT one flat /22: hosts sit on different /22s (10.13.0.x, 10.13.2.x, 10.13.7.x, 10.13.8.x) reached via `10.0.0.0/8 via 10.13.4.1`, delivered as DHCP option 121. A guest that ignores 121 has no route off its /22 | G dhcp.rs:142-165 | M 2026-09-20 | – | always, plus fault `GuestIgnoresDhcpOption121` | 29 |
| L88 | Utility and public IPs change on every re-lay | mem: gunnar-estate-2026-09-21 | M | – | always, from RFC 5737 TEST-NET only (192.0.2/24, 198.51.100/24, 203.0.113/24). The pool used to be REAL UpCloud addresses, one of them a stranger's host with port 22 open, so a mock step that connected to a handed-out address (verify, ssh to the front, the banner check) reached the internet. TEST-NET is not routed; `estate::address_tests` holds every address the estate hands out to it | 12 |
| L89 | The utility net offers option 3 and may offer a /0 via 121; accepting either gives a second default route. The ACK carries option 6 (DNS) | G pid1.rs:1308-1362; G dhcp.rs:40-45 | M/R | – | desc (the `DhcpOffer` carries 121 and no router) | 29 |
| L90 | **No hairpin NAT** | – | M | – | always | 30 |
| L91 | **IPv6**: UpCloud offers it (price 0), and each interface picks `ip_address_family`. "No IPv6" is this estate's LAW, not a provider limit | PHO main.tf:131; PGO main.tf:48; mem: korp-start-initrd-devtmpfs | R | – | **wrong**: the mock renders IPv4 whatever the body asks, so a declaration that asked for IPv6 would pass here unseen | +61 |
| L92 | **Firewall off, or a server with no rules, is wide open.** Rules are evaluated top-down, first match wins; that is why every rule set ends in a catch-all drop | PHO terraform_gate.rs:635-649; PGO reimage.rs:1228-1250 | R | – | **no**: the mock stores rules and evaluates none | +62 |
| L93 | The catch-all inbound drop also drops the REPLIES to outbound UDP (DNS, NTP); TCP replies pass. `dig +tcp` answers, `+notcp` times out | G gunnar-clock/src/lib.rs:33-47; mem: gunnar-clocks-front-is-root | M 2026-09-14 | – | fault `UdpInboundDropped`, default ON, **everywhere**. It is really a consequence of the rules (L92); a server with firewall off gets its UDP replies | 18, +62 |
| L94 | The firewall also filters the UTILITY interface | PGO terraform/main.tf:476-489 | M 2026-09-13 | – | **no** | +62 |
| L95 | A DROP is silence until the caller's timeout; an admitted port with no listener answers RST | PGO live.rs:7963-7975 | M 2026-09-20 | knock 5 s | partly: `Reach::Refused` exists, a drop does not | +62 |
| L96 | New servers came up `started` as bare Ubuntu with TCP/22 open | PGO CUTOVER-2026-09-13.md:106-107 | M | – | always (TemplateFirstBoot) | 21 |
| L97 | First boot of the Ubuntu template: `unattended-upgrade-shutdown` holds `dpkg lock-frontend`; sshd may refuse for a few seconds after apt | lib.rs #21; mem: gunnar-estate-3vm-2026-09-15 | M | a few s | always / desc | 21 |
## 9. The terraform provider (UpCloudLtd/upcloud 5.44.1) against the mock
MEASURED by this lane on 2026-09-21. The real provider binary (built with
go1.25.3) ran through a full cycle for both estates' declarations: `init`,
`apply`, `plan` (no changes), `plan` with each variable perturbed, `apply` of
growths, and `destroy`. The mock ran at `--speed 20000 --log`.
**The request contract.** These are the only calls 5.44.1 sends. Both estates
send the same set. Zero `MOCK_UPCLOUD_NOT_IMPLEMENTED`.
| call | gunnar cycle | holger cycle | notes |
|---|---|---|---|
| `GET /1.3/account` | 12 | 12 | one per provider configure; the credential probe |
| `GET /1.3/plan` | 12 | 12 | before every server read and write |
| `GET /1.3/zone` | 12 | 12 | likewise |
| `GET /1.3/storage/template` | 3 | 3 | template title → uuid, once per server create |
| `POST /1.3/server` | 3 | 3 | the create carries interfaces, login, template, devices |
| `GET /1.3/server/{uuid}` | 22 | 24 | refresh, and the provider's own polling |
| `POST /1.3/server/{uuid}/stop` | 3 | 3 | destroy stops before it deletes |
| `DELETE /1.3/server/{uuid}` | 3 | 3 | never with `?storages` |
| `GET /1.3/server/{uuid}/firewall_rule` | 6 | 6 | |
| `PUT /1.3/server/{uuid}/firewall_rule` | 6 | 6 | the whole rule set in one call |
| `POST /1.3/storage` | 5 | 2 | |
| `GET /1.3/storage/{uuid}` | 20 | 8 | |
| `DELETE /1.3/storage/{uuid}` | 8 | 5 | the template disk is deleted separately, right after the server DELETE |
| growth only: `PUT /1.3/server/{uuid}`, `POST …/start`, `PUT /1.3/storage/{uuid}`, `POST /1.3/storage/{uuid}/resize` | yes | – | plan change; template grow with autoresize; data volume grow |
**Provider behaviours, measured through the mock.**
| # | What the provider does | M/R | mock |
|---|---|---|---|
| L98 | `tier` on `upcloud_storage` is ForceNew (maxiops → hdd: "must be replaced") | M (5.44.1) | n/a (provider) |
| L99 | `login` on `upcloud_server` is ForceNew (a changed key replaces the front, and its firewall rules follow via `server_id`) | M (5.44.1); PHO adopt.tf:21-28 | n/a |
| L100 | In place: plan, template size (**both up and down in the plan**, so a shrink is only refused at apply by the API), data volume size, hostname, labels. `boot_order` is ignored by `ignore_changes` | M (5.44.1) | n/a |
| L101 | **A data volume grow on a RUNNING server is sent straight through**: the provider does NOT stop the server for an `upcloud_storage` size change. Against UpCloud (L70) that apply fails | M (5.44.1) | the mock accepted it → +56. Re-run after: `apply -var appliance_data_disk_gb=20` fails `409 STORAGE_ATTACHED` on all four members, while plan change and template grow still apply. **The estate cannot grow its data volumes with terraform while the appliance runs** |
| L102 | A plan change: `stop` → `PUT server {plan}` → **`start` immediately, without waiting for the PUT's `maintenance` to clear** | M (5.44.1) | the mock let the start overwrite the pending transition and LOST the plan change's Resize Backup → +48 |
| L103 | A template grow with `filesystem_autoresize`: stop → PUT storage → POST resize → DELETE the `Resize Backup` → start | M (5.44.1) | always |
| L104 | Destroy: stop → DELETE server → DELETE template storage at once → DELETE data volumes. "Destroy complete" while the servers still read `maintenance` | M (5.44.1) | always |
| L105 | `~> 5.44` in both `versions.tf` will take 5.45 when it ships. Proposed: `= 5.44.1` (owner rules; see the report) | – | contract test asserts the lock |
| L106 | The provider reaches the mock only through the undocumented `UPCLOUD_DEBUG_API_BASE_URL`. If a release drops it, a "mock" run goes to `api.upcloud.com` | R | +63: the mock records which tokens it heard, so a verb can prove the provider spoke to it before `apply`/`destroy` |
| L107 | `--speed 0` breaks the provider: the mock closes a transition halfway per poll, and the provider's wall-clock budget expires first (`SERVER_STATE_ILLEGAL` on a resize). Drive it at 20 000 | M (T3) | desc (clock.rs) |
| L108 | **After the grow engine enlarges a data volume, the next plan is an IN-PLACE shrink, not a replace**: `# upcloud_storage.appliance_data[0] will be updated in-place` / `~ size = 20 -> 10`, and the apply is refused `400 SIZE_INVALID` per volume, exit 1, nothing destroyed. **Every later apply then exits 1**, whatever else it carries. The fix, `lifecycle { ignore_changes = [size] }`, gives "No changes". A replace of the data still comes from `tier`, `zone` or a lower `count`; only `prevent_destroy` stops those. Both estates were checked | M (5.44.1 through the mock, 2026-09-21) | n/a (provider plus declaration) |
| L109 | A server PUT changes `hostname`/`title` in place | M (5.44.1) | **was dropped** by the mock, so terraform refused its own apply → 64 |
| L110 | `POST /server/{uuid}/cdrom/load` `{"storage_device":{"storage":…}}` puts a medium into the existing cdrom device; with CD first, the next hypervisor start boots it. Refused into a full tray (`409 CDROM_DEVICE_IN_USE`, eject first), with no cdrom device (`404 CDROM_DEVICE_NOT_FOUND`), for a storage not `online` (`409 STORAGE_STATE_ILLEGAL`). The provider's `type: cdrom` storage requirement is not enforced (L49 attaches `normal` uploads as cdrom) | R (UpCloud 1.3 docs; **not measured**) | always → 65 |
| L111 | **An import straight after `POST /storage` is accepted**, with no wait for `online`. gunnar `deploy/upcloud/src/plan.rs:550-553` (`start_medium`) calls create and then import with no wait and no retry (`api.rs:350,364-378`), and one refusal fails the run. The live re-image `private-gunnar-ops/.reimage/reinstall-receipt.json` (2026-09-21 08:42) logged `media-cdrom` at 24 504 ms and `media-virtio` at 47 439 ms, with `done: true`. The 2026-09-20 clone probe (`tests/clone_probe.rs:160-167`) did the same. **Unverified: what state the new storage READ at that moment.** The 3 s `maintenance` after a create is the mock's own number, never measured | M (acceptance) / R (state) | **was refused** `409 STORAGE_STATE_ILLEGAL` by the mock, so a mock re-image stopped at the first import → 66 |
| L112 | **Which spelling of a number in a request body is VERIFIED against the account** (lane T14, 2026-09-21). `PUT /storage/{uuid}` `size`: **the JSON NUMBER is verified** — monetize-impl's grow ran live on 2026-09-14 with `{"storage":{"size":<number>}}` and a measured 37 s to `online` (`monetize-cloud-impl/src/upcloud/grow.rs:78`); UpCloud's SDK declares it `int`; provider 5.44.1 refuses a string (T7). The STRING was never sent to the account — private-gunnar-ops' "a JSON number is refused" came from `30979d4` ("the shapes are asserted, never sent") and is withdrawn; its grow now sends the number. `POST /server/{uuid}/stop` `timeout`: **the STRING is verified** — sent live by gunnar-upcloud (every re-image since 2026-09-07) and monetize-impl (`"60"`); the NUMBER is UNVERIFIED (gunnar-upcloud's "accepted and ignored" names no run). `PUT /server/{uuid}` plan: a name, no number. | M (size: number; timeout: string) / R (the other spelling of each) | the mock ACCEPTS both spellings of both — it must not pretend to know the unverified half → 68 |
| L113 | **A REBOOTING medium** (composed without `install_then=poweroff`; every korp-installer < 0.1.4; the 2026-09-14 loop): its installer start reads **`started`** for the whole pass, 900–1100 s, and the pass ends in a guest reboot — the CD still loaded and first installs again (17), otherwise the disk boots. `cdrom/eject` answers 200 on the started box mid-pass | G deploy/upcloud/src/plan.rs (every measured pass ≥ 900 s); live 2026-09-08 (1000–1100 s); eject on a started box MEASURED 2026-09-14 (mem: upcloud-reimage-install-loop) | M | 900–1100 s | fault `installer-reboots` (by name: a property of the medium); disarmed is the power-off medium (L33) | 69 |
## 10. Timing table, provider vs guest
| event | UpCloud (outside) | guest (inside) | mock `Timings` before | after |
|---|---|---|---|---|
| server create | 98–105 s | – | 98–105 s | same |
| start → started (disk boot) | ~10 s | – | 20 s | **10 s** |
| start with an installer that powers off | `started` at ~3.6 s for < 100 ms, then `maintenance`, `stopped` 130–154 s after the start (2026-09-21 receipt; 2026-09-19 polled too coarsely to see the `started`) | install 2–5 s (UART), 10 s KVM | `started` 3.6–3.7 s, `maintenance`, `stopped` 130–146 s | **the brief `started` is real; a waiter must accept `stopped` after `maintenance`** |
| re-image wall (ladder) | 110–253 s (3 runs); older, heavier runs 1000–2865 s | 2–5 s | observers only | same |
| storage create | – | – | 3 s | same (unmeasured) |
| import upload 43 MB | 1–5 s | – | 120 ms/MiB | same |
| import sync | 100–130 s after "completed" | – | 100–130 s | same, + trailing `maintenance` |
| clone | 728 ms call, 47 s to online, no sync | – | 3 s + 100–130 s sync | **47 s, no sync** |
| detached grow | 37 s back to online | – | instant | **37 s** |
| delete front / 4-member appliance | ~60 s / >300 s | – | 60 s + 65 s/vol | same |
| VNC toggle settle | ~2 s | – | 0 | **2 s** |
## Misunderstandings: things once believed about UpCloud that were false
The mock must not encode these. Each one names what the mock did about it.
| # | The belief | What is true | Mock |
|---|---|---|---|
| X1 | The API sends no `created`, so a young volume looks like an orphan | `created` is on the list and the detail (MEASURED 2026-09-20) | corrected (behaviour 10); a hypothesis fault remains |
| X2 | "The boot disk is the smaller one" / "class then largest" is safe | No size rule serves both the appliance (50+4×32) and the twin (10+20). It is fixed by declaring `korp.data.disks` | n/a (installer), but see +58 on addresses |
| X3 | A clone syncs like an import, so it only saves the 5 s upload | A clone skips `syncing`: 47 s against 114 s (MEASURED). But it is serial after its source, so two parallel imports win | **the mock still encoded the belief** → fixed, +53 |
| X4 | A virtio detach on a running server answers a 409 `*_HOTPLUG_UNSUPPORTED` | It answers `511 HOTPLUG_FAILED` when the guest cannot hot-plug. monetize-impl only catches the 409, so a real 511 maps to `Unavailable` and skips its fallback | **the mock encoded the invented 409** → fixed, +60 |
| X5 | Outbound UDP is blocked | The inbound catch-all drops the REPLIES | desc → +62 |
| X6 | Setting `boot_order=disk` by API stops an install loop | Honoured only at the next hypervisor start; eject is the primitive | correct already; now tested (+49) |
| X7 | `stop_server` takes `timeout_action` | 400; it belongs to restart | +47 |
| X8 | `started` means healthy | It says nothing about the guest (xfs brick read `started`) | always |
| X9 | `stopped` means off | hentor boxes still answered next day | desc |
| X10 | A re-image is done when `started` is read | A powering-off installer never reads `started`: maintenance → stopped | **the mock encoded the belief** → +46 |
| X11 | An IDE cdrom needs `CONFIG_ATA` | A donor kernel with ATA off booted (the ISO rides virtio for Linux) | n/a |
| X12 | UpCloud has no UART | Only ttyS0 is missing; ttyS1/ttyS2 exist | desc |
| X13 | The utility network is one flat /22 | Several /22s joined by an option-121 route | correct (29) |
| X14 | The utility network is unfiltered | The firewall filters it too | +62 |
| X15 | Detach gives the storage back | Detached storage still bills | n/a |
| X16 | A present credential is a valid one | A dead token started cleanly and failed a day later at the first price read | always (2) |
| X17 | `412 out_of_stock` and "0 rows + 403" are UpCloud behaviours | Both dated measurements are **Scaleway** (fr-par-1 2026-09-09; IAM 2026-08-26). UpCloud's dead token is a clean 401 (MEASURED) | **relabelled, not removed** (owner ruling 2026-09-21): faults 3/37 and 5 are marked "verified against Scaleway, not UpCloud; kept as a generic cloud-provider fault" in `lib.rs` and `faults.rs`; they stay armable and in the seeded weather, so no storm seed changes |
| X18 | Resizes are batched in a 10 s window by UpCloud | The estate batches; UpCloud resizes each call | n/a, the mock must not batch |
| X19 | Resize while attached works (terraform has no stop) | Refused while the server runs (docs, both readings) | **the mock let it through** → +56 |
| X20 | The ladder's `install-time` of 110–253 s is the installer | It is the provider's wall time; the installer is 2–5 s | correct (32) |
| X21 | The mock's 38 behaviours were all measured on UpCloud | 3, 5 and 37 are Scaleway; 26's "not measured" was measured; 16's virtio code was invented | this ledger |
| X22 | A plan change mints the 44 GB Resize Backup | The measured 44 GB came from the twin's FILESYSTEM resize. Plan change → backup is unconfirmed | 9 kept as-is, labelled unconfirmed |
| X23 | Reads are consistent with writes | A resize answers 200 while the size still reads old; an attach answers 200 while it still reads detached (reported) | partly (+55 gives the resize a `maintenance` window) |
| X24 | `terraform destroy` "complete" means the servers are gone | They are still `maintenance` for minutes | always |
## Counts
- Ledger rows: **107** behaviours (L1–L107) and **24** misunderstandings (X1–X24).
- Already in the mock and right: about 67 (55 always-on or answered by construction, 12 armable faults).
- Described only (the virtual guest cannot show it; T12's real guest can): 7.
- Not the provider's behaviour, so the mock must not carry it (n/a): 4, plus the three ForceNew rows, which belong to the provider binary.
- **Missing or wrong: 25 rows, closed as behaviours 39–63** (below).
- Verified against Scaleway, not UpCloud; kept as generic cloud-provider faults (owner ruling 2026-09-21): behaviours 3, 5 and 37.
- Left open on purpose (unmeasured, or the error code is unknown): L10 key rules, L12 (201 vs 202 on create), L19, L26, L53, L61, L64, L71.
## Added by lane T13 (see `lib.rs`)
| # | behaviour | kind |
|---|---|---|
| 39 | an empty path segment (`/1.3//storage`) is 404, as UpCloud answers | always |
| 40 | a read answers 502 | fault `ReadBadGateway` |
| 41 | `PUT /server/{uuid}` answers 202 | always |
| 42 | a chunked upload answers 411 | always |
| 43 | a dead token is a clean 401 everywhere | fault `DeadToken` |
| 44 | `403 MAXIOPS_STORAGE_LIMIT_REACHED` on a create/grow past the quota | fault `StorageLimitReached` |
| 45 | start → started in 10 s | always (timing) |
| 46 | a guest that powers off inside the start never reads `started` | always (the guest decides; the clock gives the floor) |
| 47 | `stop_server` with `timeout_action` → 400 | always |
| 48 | a `start` during a plan change's `maintenance` is refused 409, and the Resize Backup is not lost | always |
| 49 | a PUT of `boot_order` does not stop a looping guest; eject does | always (test) |
| 50 | the VNC toggle settles after 2 s; read too soon, the old port comes back | always |
| 51 | a PUT to a server in `maintenance` → 409 | always |
| 52 | the import's storage goes syncing → maintenance → online | always |
| 53 | a clone is maintenance → online in 47 s, no sync | always (measured); `CloneSkipsSync` is kept as a no-op alias |
| 54 | shrink **and same size** → `400 SIZE_INVALID` | always |
| 55 | a size grow puts the storage in `maintenance` for 37 s | always |
| 56 | a size grow of a storage attached to a non-stopped server → 409 | always; fault `ResizeRequiresDetach` for the stricter reading |
| 57 | attach of an already-attached storage → 409 `STORAGE_ATTACHED` | always |
| 58 | a requested address is honoured; `virtio` picks the first free slot; guest names follow the slots (vda = virtio:0) | always |
| 59 | delete of a storage in `maintenance`, or of a clone's source mid-clone → 409 `STORAGE_STATE_ILLEGAL` | always |
| 60 | virtio hot-plug on a started server is the GUEST's: a hot-plug kernel succeeds; without one → `511 HOTPLUG_FAILED` | fault `GuestKernelLacksHotplug` (per server at create) |
| 61 | an interface asking `IPv6` gets an IPv6 address | always |
| 62 | the firewall evaluates: off or no rules = open; top-down; drop = silence; utility filtered too; UDP replies are dropped only by a rule set that ends in a drop | always (`net::firewall_admits`) |
| 63 | the mock records the tokens it heard (by digest), so a verb can prove the provider spoke to IT before `apply`/`destroy` | `/mock/heard` |