Module topology 

R850 — what the declared topology does when a node dies, and whether it fits on the boxes it is declared against.

The question this module exists to answer

From the noisetable camp, 2026-09-02: “I have a singleton process with three in-process turso DBs on one node. I kill that node at the hardware level. What happens?”

Every fact needed to answer that is already in .yah/ — the archetype, the volume sources, the replica count, the restart policy, the node’s taints and [allocatable], the sovereign role. What was missing was anywhere they were read together, so the honest answer required reading yubaba’s source. This module is that reading, done once, as a pure function.

Pure, declaration-only, and read-only

analyze takes a CloudConfig and returns a Topology. No network, no credentials, no filesystem beyond the load that already happened, and nothing here changes runtime behaviour — the same contract crate::migrate::plan_migration holds, and for the same reason: an answer you can only get by probing a live fleet is an answer you cannot get before committing the topology, which is exactly when it is worth having.

The cost of that purity is that placement here is projected, not observed: Placement is what the admission seam (CloudConfig::admit_workload_candidates) would decide today, not where containers are running right now. Where the two can differ is stated on Placement itself.

What it deliberately does not model

• Liveness. A node declared here may be off. Admission has no liveness input by design (see admit_workload_candidates), and neither does this.
• Correlated failure. “Node X dies” is one node. Losing a rack, a region, or the upstream the whole camp NATs through is a different question with a different answer, and pretending a per-node walk covers it would be worse than not asking.
• The blast radius of the control plane itself. Losing the box that hosts the operator bridge is modelled only as far as QuorumEffect goes.

@arch:see(.yah/docs/working/W305-sovereign-groups-environments-edges.md)

@yah:ticket(R850-F1, “Hydrate-on-place: act on the declared durability tier at runtime, with fencing”) @yah:at(2026-09-05T09:34:40Z) @yah:assignee(agent:bundle-anthropic-ashguard) @yah:phase(P4b) @yah:parent(R850) @yah:next(“R850-P4 landed the DECLARATION half: yah.durability.{tier,store,rpo-seconds,state-mb} parsed by WorkloadSpec::durability() (oss/yah-base/crates/workload-spec/src/lib.rs), hard-validated in validate::shape, and read by cloud::topology. Declaring a tier still causes NO backup and NO restore. This ticket is the runtime half.”) @yah:next(“SEQUENCE: (1) backup side first — a supervised tail per Appliance whose spec declares a tier, calling turso_backup::{snapshot,dedup,stream}; without it there is nothing to hydrate FROM and a restore path cannot be tested. (2) hydrate-on-place: before kamaji starts a container whose named volume is EMPTY at /var/lib/yah/kamaji/volumes/<name>, restore from the declared store. Empty-vs-populated is the trigger, so a normal restart never re-hydrates.”) @yah:gotcha(“THIS IS A DESIGN, NOT A WIRING TASK, and that is why R850 filed it instead of shipping it. Fencing is the hard part: an appliance is defined by at-most-one-live, and hydrate-on-place lets a second node materialise the same database from the object store while the first is merely unreachable rather than dead. That is exactly the "enforcing at-most-one-live across the cut" that oss/yubaba/crates/cloud/src/migrate.rs’s module header deferred to its own relay. turso-backup already has two-level fencing in stream.rs — read it before inventing one.”) @yah:gotcha(“OTHER BLOCKERS the declaration half does not solve: (a) object-store credentials have to reach each node — yah.durability.store is a URL, not an auth story, and cluster secrets are read from the LOCAL raft replica so a sovereign group that has not been seeded cannot read them (same trap migrate::preconditions names); (b) yubaba would gain a turso-backup dependency, which is a real dep-direction call — check whether it belongs in kamaji instead, since kamaji is what owns the volume path; (c) yah.durability.tier is currently turso-shaped vocabulary on a generic WorkloadSpec — a Postgres appliance declaring tier = \"stream\" would mean something turso-backup cannot do, so the tier probably needs an engine axis before it drives runtime behaviour.”) @yah:verify(“yah cloud topology --kill <node> for a workload declaring a real tier must stop reporting RecoveryEstimate as an extrapolation and start reporting a measured restore. The analyzer is already the acceptance surface — oss/yubaba/crates/cloud/src/topology.rs, test hardware_killing_the_node_under_a_singleton_appliance_loses_everything.”) @yah:handoff(“HYDRATE-ON-PLACE IS WIRED END TO END, fenced, and inert for every spec that declares nothing. Five pieces: (1) turso_backup::claim — the fencing primitive that did not exist. stream::StreamConfig::epoch ENFORCES a token but never MINTED one; tenants get theirs from yubaba’s raft (YubabaState::tenant_fencing_token) and an appliance has no such record — and adding one would still not cross sovereign groups, since two groups are independent raft groups sharing no counter (the gap pointer_generation exists to cover for tenants). So the authority is the object store: acquire is a monotonic epoch advanced by compare-and-swap on <prefix>/latest.owner-claim, assert_holds re-verifies. Chosen because it is the SAME store the restore reads from — ‘cannot reach the fence’ and ‘cannot hydrate’ become one condition instead of two. It is deliberately NOT a lease: no TTL, no clock. Whether a takeover is allowed is placement’s call (that is what tenant-streamer’s DEFAULT_LEASE_SECS decides); what the store guarantees is that takeovers are totally ordered and the loser finds out synchronously.”) @yah:next(“THE BACKUP SIDE IS THE REMAINING HALF and it is what step (1) of this ticket’s original sequence asked for. Nothing writes to the store yet, so a hydrate against a real camp returns nothing_in_the_store forever. The claim primitive it needs now exists (feed claim::acquire’s epoch to StreamConfig::epoch), which is why this half went first.”) @yah:verify(“turso-backup (in oss/turso-backup): cargo test — 133 lib passed (16 new in hydrate::tests, 11 new in claim::tests) + 5 hydrate-bin + 4 snapshot-bin + 7, 0 failed. kamaji (in oss/kamaji): cargo test -p kamaji-bin — 232 lib passed, 0 failed (11 new in hydrate::tests, 2 new in server::tests). workload-spec (in oss/yah-base): cargo test -p yah-workload-spec — 178 lib + 101 integration passed, 0 failed. Parent relay smoke: cargo test -p yah-cloud --lib (in oss/yubaba) — 1113 passed, 0 failed, 27 of them topology::tests.”) @yah:handoff(“(2) turso_backup::hydrate — the decision plus the execution. The trigger R850 filed was ‘restore when the volume is EMPTY, so a normal restart never re-hydrates’; that is right and incomplete, because a workload with three databases has a THIRD state. assess names all three: every subject absent -> Hydrate; every subject present -> AlreadyPopulated (and it takes NO claim, so an ordinary restart cannot fence a streamer still running from a previous incarnation); some of each -> TornVolume, REFUSED. Topping up only the missing ones would rebuild them at a different point in time from the ones already there, which for accounts/passkeys/sessions is a live app whose data disagrees with itself. Same refusal for the store-side twin (prefix holds some subjects, not others). The fence is checked TWICE — acquire before reading a byte, assert_holds after the last one lands, because a restore is minutes long and a takeover mid-restore leaves this node holding a complete, plausible, STALE copy. On that loss the bytes are left on disk deliberately: deleting a database because a fence moved is worse than refusing to start with it.”) @yah:handoff(“(3) DEP DIRECTION RESOLVED — gotcha (b). Neither yubaba nor kamaji links turso-backup. New bin turso-backup-hydrate (oss/turso-backup/src/bin/hydrate.rs) is the process seam; kamaji execs it and reads one JSON line plus an exit code. Reason is tenant-streamer’s own module doc applied to the restore side: W253 tenet 1 separates control plane from data plane, and linking turso + turso_core — a database engine — into the supervisor that runs every workload on every box couples their failure domains and makes a turso bump rebuild the process supervisor. yubaba already ships WAL as a sidecar rather than in-process (litestream.rs, then tenant-streamer); this is that shape. Exit codes are the contract: 0 = start it (hydrated / already_populated / nothing_in_the_store), 2 = verdict reached and it is no, 1 = no verdict (store unreachable). 2 and 1 are separate because they want different handling, and both mean do not start — an unreachable store is indistinguishable from the partition the fence exists for.”) @yah:handoff(“(4) ENGINE + SUBJECT AXES — gotcha (c) closed. yah.durability.engine (turso; anything else is a hard UnknownEngine) and yah.durability.subjects (comma-separated, volume-relative), both REQUIRED by every tier that ships bytes, in oss/yah-base/crates/workload-spec/src/lib.rs. Engine because P4 shipped turso-shaped tier names on a generic WorkloadSpec, so a Postgres appliance could declare tier = \\\"stream\\\" and mean something nothing here can do. Subjects because a restore’s unit is a FILE and a workload’s is a VOLUME — the driving case is three turso DBs in one named volume, ‘restore the volume’ is not a thing turso-backup can do, and guessing which files in a directory are databases is guessing about the only copy of somebody’s data. Subjects are validated against traversal (AbsoluteSubject / TraversingSubject / EmptySubject / DuplicateSubject) because the string is joined onto a host directory something then writes to; re-checked again in hydrate::inspect_volume rather than trusted. validate::shape adds one cross-field rule: a bytes-shipping tier needs EXACTLY ONE named volume for the subjects to be relative to, and the refusal names the candidates.”) @yah:handoff(“(5) KAMAJI HOOK — oss/kamaji/crates/kamaji-bin/src/hydrate.rs + the call in deploy_container (server.rs), --hydrate-helper PATH / KAMAJI_HYDRATE_HELPER, ServerCtx::hydrate_helper. Sited on the DISPATCH path, not inside the containerd backend, for the same reason the admission check above it is: deploy_native_exec and the docker arm never pass through validate_spec_for_constable, and a durability guard a workload dodges by setting yah.exec = native is not a guard. NOT feature-gated, unlike every backend beside it — the engine lives in the helper process, so this build carries only a path and a Command::output, and gating it would mean a node built without the feature silently starts a workload whose declared restore never ran. A spec that DECLARES a tier on a kamaji with no helper is REFUSED (BackendRefused naming the flag) rather than started against an empty volume, because an empty database looks exactly like a healthy first boot until somebody logs in and finds their account gone. Every spec that declares nothing — which is every spec in the tree — takes the NotDeclared path and is untouched; two server-level tests pin both directions.”) @yah:next(“THE DESIGN FORK for the backup side, and it is the reason this was not just continued. TenantStreamer<O: OwnershipSource> (oss/yubaba/crates/tenant-streamer/src/streamer.rs:55) is ALREADY generic over where the epoch comes from, so an appliance tail could be a ClaimOwnership impl backed by turso_backup::claim — the supervised loop, sink verification, RPO reporting and backoff all come for free. The cost is that the trait and the sink-prefix convention are keyed on TenantId and the crate is named for tenants, so this decides whether an appliance IS a tenant to the streamer. (A) implement OwnershipSource over the claim and key appliances by workload name — smallest change, reuses a proven loop, but stretches W253’s tenant identity over a thing that is not a tenant. (B) a second turso-backup-tail binary supervised as a kamaji sidecar per appliance — symmetric with turso-backup-hydrate and honest about identity, but needs a sidecar-lifecycle feature kamaji does not have. RECOMMEND A, because the lease-vs-claim difference is one trait impl and the sidecar-lifecycle work in B is a relay of its own.”) @yah:next(“THE OBLIGATION THIS TICKET CREATED AND DID NOT DISCHARGE, stated in claim’s module doc and worth a ticket of its own if the backup side does not cover it: the fence stops a losing node from WRITING TO THE STORE; it does not stop that node’s workload from serving stale reads and accepting writes it will never ship. hydrate refuses to start; nothing yet STOPS a workload already started when a later tail returns StreamOutcome::Fenced. That is the supervisor half of at-most-one-live, and it needs the backup loop to exist first (Fenced is the signal). Whichever fork above is taken must wire Fenced -> kamaji Stop.”) @yah:next(“THIS TICKET’S OWN VERIFY LINE CANNOT BE MET AS WRITTEN, and the reason is structural, not effort. It asks that yah cloud topology --kill <node> ‘stop reporting RecoveryEstimate as an extrapolation and start reporting a measured restore’. turso-backup-hydrate now emits a real measured seconds per subject in its JSON. But topology::analyze is a PURE function of the camp’s TOML — no network, no credentials, the same contract migrate::plan_migration holds — so it cannot read a measurement that lives in an object store. Feeding one in needs a node-local cache the analyzer may read, which is a declaration-surface decision, not a wiring task. Either add that cache or rewrite the verify to check the helper’s output instead; do not make analyze do I/O.”) @yah:gotcha(“CREDENTIALS — gotcha (a) is only PARTLY answered. yah.durability.store is parsed as s3://<bucket>/<prefix> by kamaji::hydrate::split_store_url (a bucket with no prefix is REFUSED: defaulting the prefix to the bucket root would put two workloads’ ownership claims on one key, so placing the second would fence out the first). Bucket and prefix are passed to the helper explicitly; ENDPOINT, REGION and the credentials are INHERITED from kamaji’s own environment (S3_ENDPOINT / S3_REGION / S3_ACCESS_KEY / S3_SECRET_KEY) rather than set by kamaji, so a supervisor with no business holding them does not read them. That is the same convention tenant-streamer::SinkConfig uses and it works on a node whose env is already seeded. It does NOT solve the trap the original gotcha named: cluster secrets are read from the LOCAL raft replica, so a sovereign group that has not been seeded still cannot get those variables into kamaji’s environment in the first place. Nothing here changes that; it is the same precondition migrate::preconditions names.”) @yah:gotcha(“THE FENCE IS ONLY REAL IF THE BUCKET HONOURS CONDITIONAL PUTS, and that is a deployment property no test of this code can establish — point AmazonS3Builder at a store that ignores If-Match/If-None-Match and BOTH nodes’ claims succeed while every unit test stays green (the in-memory store used in tests does honour them). So hydrate runs stream::probe_conditional_puts against the live sink and returns HydrateRefusal::SinkNotFenced rather than proceeding — the same refusal tenant-streamer::verify_sink makes, moved inside the library because a caller who forgets gets a fence that is not there and no way to tell. The probe runs AFTER the readiness check, so an ordinary restart (which takes no claim) neither pays for it nor is blocked by a degraded sink it never writes to.”) @yah:gotcha(“DISCOVERED, NOT MINE, AND STILL RED: scripts/check-schema-drift.sh fails on an uncommitted regeneration of .yah/schema/{workload,machine}.toml.schema.json. Confirmed NOT caused by this ticket — grepping the drift diff for ‘durability’ returns 0 lines, and the new types (Durability, DurabilityEngine, DurabilityTier) carry no TS/JsonSchema derive while WorkloadSpec’s fields are untouched. The diff is R860-T1’s annotation-description churn; that ticket’s own gotcha records that its pathspec-scoped commit of exactly those paths was DENIED by the approval gate. scripts/check-workload-spec-ts.sh is green. Nothing to regenerate here — this needs the commit R860-T1 asked for.”) @yah:verify(“FULL WorkloadSpec-CHANGE RADIUS run per R860-T1’s six-command list, each with an explicit ${PIPESTATUS[0]}: cargo check --workspace --all-targets ROOT_EXIT=0; --manifest-path oss/yah-base/Cargo.toml --all-targets YAHBASE_EXIT=0; --manifest-path oss/yubaba/Cargo.toml --all-targets YUBABA_EXIT=0; --manifest-path oss/kamaji/Cargo.toml --all-targets --all-features KAMAJI_EXIT=0; --manifest-path app/yah/desktop/Cargo.toml --no-default-features DESKTOP_EXIT=0. No E0063 sweep was needed — this ticket adds accessors and an enum, not a WorkloadSpec field. Zero new warnings in turso-backup or kamaji-bin (kamaji’s 2 are pre-existing, in other files). scripts/check-workload-spec-ts.sh: ok, index.ts in sync.”) @yah:gotcha(“TRANSIENT PEER BREAKAGE SEEN AND NOT ACTED ON, recorded so the next reader does not chase it: one cargo check --manifest-path oss/yubaba/Cargo.toml --all-targets returned YUBABA_EXIT=101 with six E0061 ‘takes 3 arguments but 2 were supplied’. The immediate re-run was clean with no edit from me. @Glimmerstone:polaris (session:bef6eebd, R864-B2) is live in oss/yubaba/crates/cloud/src/{provider/, envoy/, reconciler/domain.rs} and the build-input watcher named reconciler/domain.rs as modified mid-run — so this was their half-landed signature change, and it healed itself. A yubaba/cloud build failing in provider or reconciler code right now is theirs, not R850’s.”) @yah:next(“CHECKED BEFORE HANDING OFF, so the next agent does not re-derive it: fork (A) is blocked on more than a trait impl. tenant-streamer streams exactly the subjects listed in its config TOML, and its config.rs says so explicitly — ‘Placement. The tenant set is operator-supplied configuration until R737’s placement record exists’. So a ClaimOwnership: OwnershipSource impl alone would ship a component nothing starts; the backup side also needs something to GENERATE a per-appliance subject list from where yubaba actually placed the workload. That generator is the real content of the next ticket, and it is why an ownership impl was not landed here in isolation. Take the fork decision and the config-source decision together.”) @yah:handoff(“FILES: new oss/turso-backup/src/{claim.rs, hydrate.rs, bin/hydrate.rs} + Cargo.toml [[bin]] + two lib.rs mod lines; new oss/kamaji/crates/kamaji-bin/src/hydrate.rs + lib.rs mod line + server.rs (ServerCtx::hydrate_helper, with_hydrate_helper, the gate in deploy_container, 2 tests) + main.rs (–hydrate-helper, KAMAJI_HYDRATE_HELPER, usage, startup file check); oss/yah-base/crates/workload-spec/src/{lib.rs, validate.rs} + tests/shape_fixtures.rs; oss/yubaba/crates/cloud/src/topology.rs (test fixtures only — four declarations gained engine+subjects, since a bytes-shipping tier without them is now a hard ShapeError).”) @yah:handoff(“Tree anchor at handoff: f086233d6b092de2f32cafad5e0010494078269c — the shared tree as I left it. Diff against it (git diff f086233d6b092de2f32cafad5e0010494078269c..HEAD) to see what landed under you, and quote this SHA rather than ‘HEAD’ in any revert/restore instruction.”)

Structs

MachineNode

A declared machine, plus what the admission seam projects onto it.

NodeLoss

Everything that follows from losing one machine outright.

Oversubscription

A machine whose declared capacity does not cover what is projected onto it.

Topology

The declared fleet, read as a graph, plus every verdict derivable from it.

WorkloadImpact

What a node loss does to one workload projected onto it.

WorkloadNode

One workload, everything about it that bears on survival, and where it goes.

Enums

DataLoss

How much of the workload’s state is gone, and how far back the copy is.

DurabilityView

A workload’s yah.durability.* declaration as the analyzer sees it.

Outcome

Whether the workload comes back, and who brings it back.

Placement

Where the admission seam would put a workload, and what else could take it.

QuorumEffect

What losing a machine does to its sovereign group’s raft quorum.

RecoveryEstimate

How long it takes to get the state back, and on what basis that is claimed.

Constants

DEFAULT_STREAM_RPO_SECONDS

turso_backup::stream::DEFAULT_RPO_TARGET, in seconds.

MEASURED_GET_RTT_MS

Per-GET round-trip time, milliseconds. Measured in the same R760-T10 run.

MEASURED_HYDRATE_MB_PER_S

Bulk object-store throughput, MB/s.

Functions

analyze

Walk the declared graph once and answer every question derivable from it.