Expand description
Headscale API client and configuration helpers for yah mesh operations.
The mesh layer manages the Headscale coordinator that all yah-provisioned machines join via Tailscale. Phase 1a runs Headscale on the operator’s camp (bootstrap coordinator); Phase 1b promotes it to a cluster machine (R040-F19); Phase 2 adds openraft-based HA (R040-F20/F21).
Consumers:
app/yah/cli/src/mesh.rs—yah mesh start/status/backup/restoreapp/yah/cli/src/cloud.rs—yah cloud machine provision(auto-generates a Headscale preauth key whenmesh-urlis set in the vault)
@yah:relay(R861, “Reconcile an appliance’s own API objects: headscale ACLs, users and preauth keys as declared config”)
@yah:at(2026-09-04T21:07:28Z)
@yah:status(open)
@yah:assignee(agent:bundle-anthropic-ashguard)
@yah:gotcha(“NOT A NEW CONCEPT — this is the DOMAIN-OBJECT half of a pattern yah already has twice. Framing that produced it (operator, 2026-09-04): asked what "manage headscale on kubernetes with a rust controller" looks like, the answer is a CRD + controller for "deployment instances, users, and pre-auth keys". yah ALREADY HAS the CRD half (Workload/WorkloadSpec + workload.toml + .yah/schema/workload.toml.schema.json + generated TS) and the controller half (the Reconciler trait at oss/yubaba/crates/cloud/src/reconciler/mod.rs:612, ~20 implementations dispatched by kind). What is missing is reconciling things INSIDE an appliance rather than the appliance itself. So: no new noun, no operator framework — one more Reconciler implementation, owning headscale’s API objects the way cloudflare_worker.rs owns what lives in a Cloudflare account.”)
@yah:gotcha(“THE EVIDENCE THIS IS A LIVE GAP, not a speculative nicety. (1) headscale-preauth-key is classified Band::Automatable in oss/yah-base/crates/keys/src/spec.rs:768 and its own purpose text says outright "Automatable: headscale-api-key mints these, and does so per-machine when mesh-url is configured" — the credential system already knows it should not be a human-held static secret. (2) That automation EXISTS but only ad hoc, in one place, at one moment: provision-time per-machine minting via headscale-api-key. There is no reconciled desired state. (3) acls.yaml (77 bytes, mtime Jun 22) sits in /var/lib/yah-cloud/headscale/ and is replicated by NOTHING — litestream carries only headscale.db — so it is failover state nobody enumerated (see R858-T2). Declaring ACLs and reconciling them is what stops that file mattering at all.”)
@yah:ticket(R861-T1, “Establish whether headscale reads acls.yaml or the DB, then declare ACLs as reconciled config”)
@yah:status(review)
@yah:at(2026-09-04T22:48:06Z)
@yah:assignee(agent:bundle-anthropic-glimmerstone)
@yah:parent(R861)
@yah:next(“THE PATTERN TO COPY IS cloudflare_worker.rs, not a new abstraction: validate declared config fully BEFORE any API call (the R330-B5 fail-fast discipline the reconciler docs already name), then list-first/idempotent writes against headscale’s admin API using the existing headscale-api-key slot. Consumers of that slot are already listed in the credential spec: oss/yubaba/crates/cloud/src/mesh.rs, app/yah/cli/src/mesh.rs, crates/yah/cloud-client/src/lib.rs.”)
@yah:gotcha(“START WITH THE FACT, NOT THE DESIGN. headscale 0.23 can source policy from a FILE or from the DATABASE depending on its policy mode, and the config block read on us-west-001 2026-09-04 showed no policy: section at all — so acls.yaml may be live or may be vestigial. grep -i policy /var/lib/yah-cloud/headscale/config.yaml settles it in one look. If vestigial: delete the file so the next reader does not re-discover the same question, and this ticket collapses to the users/preauth half. If live: it is unreplicated failover state today and declaring it is the fix.”)
@yah:gotcha(“PHASE 1 SETTLED — acls.yaml IS LIVE, NOT VESTIGIAL. Measured over SSH on us-west-001 (15.204.89.240) 2026-09-04 by @Ashguard:griffin. /var/lib/yah-cloud/headscale/config.yaml lines 32-34 read exactly policy: / mode: file / path: /var/lib/yah-cloud/headscale/acls.yaml. The earlier read that reported no policy: section was wrong — the file is 1060 bytes and the block is there. acls.yaml is 77 bytes, mtime Jun 22, and contains the permissive HuJSON default {\"acls\":[{\"action\":\"accept\",\"src\":[\"*\"],\"dst\":[\"*:*\"]}]}. In-tree agrees: ALL THREE renderers emit the same three lines — crate::mesh::generate_headscale_config (oss/yubaba/crates/cloud/src/mesh.rs), generate_remote_headscale_config (oss/yubaba/crates/yubaba/src/lib.rs:4729ff) and generate_bootstrap_headscale_config (same file, :5134ff). So file mode is the DESIRED state today, not drift. Nothing was deleted; the vestigial branch of this ticket does not apply.”)
@yah:gotcha(“POLICY MODE BOUNDS WHAT ANY RECONCILER CAN DO ABOUT ACLS — both halves measured against the live v0.23.0 coordinator 2026-09-04, not inferred. (1) GET /api/v1/policy WORKS IN FILE MODE: authenticated with the vault’s headscale-api-key against https://cloud.mesh.yah.dev it returned HTTP 200 and {"policy": "yah.”)
@yah:handoff(“LANDED: HeadscaleReconciler, kind="headscale" — one more Reconciler impl, cloudflare_worker.rs’s shape, no new noun. NEW FILE oss/yubaba/crates/cloud/src/reconciler/headscale.rs (700 lines incl. 17 tests). up() order: materialize() -> workload_kind agrees -> DeclaredHeadscale::load() validates EVERYTHING (R330-B5 fail-fast, nothing below runs until every field is checked) -> resolve base_url (declared server_url, else mesh-url/HEADSCALE_URL) + headscale-api-key/HEADSCALE_API_KEY -> users list-then-create (NEVER deletes: a headscale user owns the nodes registered under it) -> preauth keys list-then-mint-if-unsatisfied -> ACL compare-then-push. Returns RunningWorkload::adopted(WORKLOAD_KIND, role, None).with_notes(…); the minted key value never reaches a log line or a note.”)
@yah:handoff(“CLIENT SURFACE ADDED to oss/yubaba/crates/cloud/src/mesh.rs (added alongside the peer hunks already in that file; nothing reverted). New types PreauthKeyRecord (with is_usable_at) and PreauthKeyRequest. New HeadscaleClient methods: create_user (POST /api/v1/user — deliberately does NOT swallow a conflict, since headscale answers 500 for an existing name with no discriminator, so a 500 here means something other than "already there"), list_preauth_keys (GET /api/v1/preauthkey?user=), create_preauth_key_with (the general mint), get_policy (GET /api/v1/policy), set_policy (PUT /api/v1/policy). The pre-existing create_preauth_key now delegates to create_preauth_key_with with its historical fixed shape (single-use, 1 hour) so every existing caller is behaviourally untouched. Parsing split into the free fn parse_preauth_key_list, following choose_user’s precedent (this crate has no HTTP mock); entries missing id/user/key are DROPPED rather than defaulted, because a half-parsed record would read as "no matching key exists" and cause a redundant mint.”)
@yah:handoff(“DECISION I MADE, since the brief left it open: declared config lives in the component’s workload.toml under kind-specific tables ([headscale] with users / acl_policy / server_url, and [[headscale.preauth_keys]] with user / tags / reusable / ephemeral / ttl_hours / store_as), parsed ad hoc via toml::Value. This is exactly how cloudflare-worker already reads its [build] and [[bindings]] — the strong workload_spec::Workload types do not model per-kind tables, and I checked .yah/schema/workload.toml.schema.json: no branch of its root oneOf sets additionalProperties:false, so extra tables validate. CONSEQUENCE: oss/yah-base/crates/workload-spec/ was NOT touched, so NO emit-schemas and NO export-ts regen was needed or run. That also kept me clear of oss/yah-base/crates/keys/src/spec.rs, which I read (line 768’s Band::Automatable classification) but did not modify, per the leader’s read-only instruction — @Ashguard:polaris is live on it for R856-F2.”)
@yah:handoff(“THE ACL STEP IS COMPARE-FIRST, AND THAT IS THE WHOLE POINT ON A FILE-MODE COORDINATOR. get_policy() -> policies_equivalent() -> only on drift does it set_policy(). Comparison is SEMANTIC, not byte: both sides are normalized from HuJSON to serde_json::Value and compared as values, so reformatting or re-commenting the declared file never triggers a spurious push and a genuinely different rule always does. There is no HuJSON crate in this workspace, so parse_hujson strips // and /* */ comments and trailing commas with string-literal tracking (a :7091) plus its import at // or a comma inside a value is left alone — tested) and hands the rest to serde_json. On a database-mode coordinator the push lands and the policy ends up in headscale.db, which litestream already replicates — that is the moment acls.yaml stops being unreplicated failover state. On today’s file-mode coordinator the push is refused by headscale and the reconciler surfaces that as an error naming the declared file’s path and the mode flip that makes it pushable, rather than a silent no-op that reads like success.”)
@yah:handoff(“DELIBERATELY NOT DONE, and it is the one operator call left: I did NOT flip the fleet’s rendered policy.mode from file to database. It is a live-coordinator config change that reaches nodes only via a yubaba release + scripts/roll-node.sh, it touches all three renderers, and its correctness turns on what a database-mode headscale with no policy row does to an existing tailnet — which I did not measure and would not guess at fleet scale. It also sits squarely in the operator’s stated R858 rehearsal ("bring us-west-001 down and get a repaired headscale"), so it wants that rehearsal rather than a reconciler’s side effect. Until it flips, this reconciler makes ACL drift LOUD on the current file-mode box; after it flips, the same code makes ACLs actually reconciled and the file irrelevant. Nothing else about the reconciler changes either way.”)
@yah:handoff(“WIRING (all additive, no rewrites): oss/yubaba/crates/cloud/src/reconciler/mod.rs gains pub mod headscale; and a re-export of HeadscaleReconciler + DeclaredHeadscale/DeclaredPolicy/DeclaredPreauthKey + WORKLOAD_KIND as HEADSCALE_WORKLOAD_KIND. oss/yubaba/crates/cloud/src/lib.rs: ONE additive name (HeadscaleReconciler) inserted into the existing pub use reconciler::{...} list — the single-line limit the leader set for that peer-owned file (@Ashguard:spade is live on it for R850); I touched nothing else there. Both dispatchers gain a "headscale" match arm next to "cloudflare-worker": app/yah/cli/src/cloud.rs:reconcile_component (:601, and app/yah/desktop/src/mirror_run.rs (:988) plus its import at ~:168. Cleared with @Glimmerstone:vortex, whose R858-T1 courier (session:9f8108d4) is editing app/yah/cli/src/cloud.rs concurrently — my two regions do not overlap theirs (:1466, :4250-4310, :4871-4909, :7909-8006), and I told them line numbers below :7091 shift by 3.”)
@yah:verify(“cargo build -p yah-cloud: clean (the 2 warnings are pre-existing unused imports in reconciler/mesofact_static.rs:240,245, not mine). cargo test –manifest-path oss/yubaba/Cargo.toml -p yah-cloud –lib = 1059 passed / 0 failed / 4 ignored, against a pre-change baseline of 1042 passed / 0 failed on the same crate — the delta is exactly the 17 new reconciler::headscale::tests, all green. Note yah-cloud is in the oss/yubaba workspace, not the root one: cargo test -p yah-cloud from the repo root fails with "requires dev-dependencies and is not a member of the workspace"; the –manifest-path form above is the one that runs. cargo check -p desktop: clean (9 pre-existing dead-code warnings). The 17 tests cover: full workload-dir load off disk incl. the HuJSON policy file; a preauth key naming an undeclared user (the R608-B19 typo class, which must fail BEFORE any user is created); tag prefix + lowercase rules; non-positive ttl; duplicate user; non-http server_url; a policy with no top-level acls array; a missing policy file naming the path it looked for; reformat-is-not-drift vs a-different-rule-is-drift against the exact bytes live on us-west-001; comment markers inside string values surviving the stripper; and expired / spent / reusable / never-expiring key usability.”)
@yah:verify(“cargo check -p yah = exit 0, Finished clean, no error lines (warnings are the crate’s pre-existing 19: unused imports Verdict / CommandFact / NodeId / store / Version and unused variables dt / config_path / acl_path / pid_path, plus dead-code fields — none in a file I touched). Re-run once after reflowing the two use blocks, so the green describes the final tree. NOT VERIFIED, STATED PLAINLY: the reconciler has never been executed against a live coordinator — no yah cloud apply was run, no user was created, no key was minted, no policy was pushed. Every claim about headscale’s API in this ticket comes from read-only GETs against https://cloud.mesh.yah.dev with the vault’s headscale-api-key, plus string extraction from the pinned v0.23.0 binary on us-west-001. There is also no live service/mirror definition wired for kind="headscale" anywhere in .yah/services/ — deliberately, since committing one arms a reconcile against the production coordinator, and that is the operator’s call, not a courier’s. The shape is instead pinned by a test that loads a full workload dir (workload.toml + acls.hujson) off disk and asserts it is already in sync with the exact bytes the live coordinator serves.”)
@yah:handoff(“DISCOVERED WORK DONE, not filed as a followup: R858-T2’s own @yah:gotcha on oss/yubaba/crates/yubaba/src/headscale_appliance.rs asks the exact question I answered ("establish whether this file is live or vestigial … before designing carriage for it"), and leaving it open would make the next reader repeat the SSH trip. Wrote a durable @yah:notify_on(R861-T1) into R858-T2 carrying the verdict and what it means for that ticket: while the fleet stays in file mode, acls.yaml is still state a move must carry alongside noise_private.key, and it stops being carriable state only when policy.mode flips to database. That is the only edit I made outside this ticket’s own path list, and it is an additive annotation line.”)
@yah:handoff(“PHASE 1 SETTLED BY MEASUREMENT, NOT INFERENCE — acls.yaml is LIVE, not vestigial. us-west-001’s /var/lib/yah-cloud/headscale/config.yaml carries policy: / mode: file / path: /var/lib/yah-cloud/headscale/acls.yaml (read over SSH 2026-09-04), and all three in-tree renderers emit the same block. The file was therefore NOT deleted, and the ticket did NOT collapse to the users/preauth half — the ACL half is real work and was done. The ‘no policy: section’ observation in the filing gotcha was a truncated read of the config.”)
@yah:handoff(“WHAT LANDED: HeadscaleReconciler (kind = "headscale") in oss/yubaba/crates/cloud/src/reconciler/headscale.rs, built on cloudflare_worker.rs’s shape rather than any new abstraction — exactly the relay’s framing of "one more Reconciler implementation, no new noun". It reconciles three classes of headscale’s own API objects: list-then-create users, list-then-mint preauth keys, and a semantic HuJSON compare-then-push for ACLs. All declared config is validated in full BEFORE the first network call (the R330-B5 fail-fast discipline); independently confirmed at headscale.rs:463, where a preauth key naming an undeclared user is a genuine pre-network bail rather than a doc-comment claim.”)
@yah:handoff(“SUPPORTING SURFACE: five methods added to HeadscaleClient in oss/yubaba/crates/cloud/src/mesh.rs — create_user, list_preauth_keys, create_preauth_key_with, get_policy, set_policy — all against the existing headscale-api-key credential slot, no new credential. This closes the gap R861’s evidence named: headscale-preauth-key was already classified Band::Automatable (oss/yah-base/crates/keys/src/spec.rs:768) with automation that existed only ad hoc at provision time, one machine at a time, with no reconciled desired state. There is now a desired state.”)
@yah:verify(“cargo test -p yah-cloud –lib = 1060 passed / 0 failed / 4 ignored. Baseline arithmetic reconciled independently rather than taken on trust: committed baseline is 1019, +17 new tests in headscale.rs (this ticket), +24 in topology.rs (@Ashguard:spade’s R850, a separate module that landed concurrently) = 1060. The courier reported 1059 against a 1042 baseline; the off-by-one is a mid-run tree move, not a missing test. Fail-fast validation and the reconciler wiring were re-verified by a second agent reading the source, not by re-reading the courier’s claim.”)
@yah:gotcha(“cargo check -p yah is RED right now and it is NOT this ticket. Both failure snapshots land entirely in app/yah/cli/src/keys_doctor.rs — 2x E0425 ‘cannot find value decision’ at :1574 and :1583 (a half-applied local rename), plus an earlier 9-error E0308/E0599 snapshot consistent with oss/yah-base/crates/keys/src/spec.rs moving underneath it (~207 insertions on pub enum Provider, the MINT_* consts, and 8 points in CREDENTIAL_SPECS). That is @Ashguard:polaris’s live R856 work, caught mid-edit; the diagnosis was handed to them directly. Nothing in either trace names headscale, reconciler or mesh, and R861-T1 treated spec.rs as strictly read-only — verified, not assumed.”)
@yah:gotcha(“SHARED-TREE SEAM, disclosed and acknowledged: the HeadscaleReconciler re-export needed one added name in the :307), a file @Ashguard:spade holds for R850. Purely additive — nothing removed or rewritten — but inserting the name pushed MesofactStaticReconciler past the column limit and rustfmt reflowed the pair, so the diff footprint reads -1/+2 rather than +1. @Ashguard:spade was notified and confirmed no conflict: their only line in that file is pub use reconciler::{…} list in oss/yubaba/crates/cloud/src/lib.rs (pub mod topology; at ~:272, a different region. Separately, the crate has NO rustfmt.toml and is pervasively fmt-dirty (~30 files, config.rs alone has 28 sites), so this reflow adds no new gate risk.”)
@yah:gotcha(“DO NOT run rustfmt in write mode against oss/yubaba/crates/cloud/src/lib.rs. rustfmt follows mod declarations, so a write-mode run on that one file rewrites ~30 files and 100+ sites across the crate — most of it peers’ uncommitted in-flight code. That is precisely the shape of the 2026-08-28 incident recorded in CLAUDE.md, where a cargo fmt followed path deps into a peer’s crate and undoing it cost 827 uncommitted lines. Use rustfmt --check and read the file list first. Surfaced by @Ashguard:spade while checking this ticket’s reflow.”)
@yah:handoff(“NOT COMMITTED, deliberately. The shared index currently reads as staged by someone other than this ticket’s worker, so nothing here was committed — a bare git commit on this tree would sweep in peers’ work. Whoever signs this off should commit pathspec-scoped: oss/yubaba/crates/cloud/src/reconciler/headscale.rs, oss/yubaba/crates/cloud/src/reconciler/mod.rs, oss/yubaba/crates/cloud/src/mesh.rs, and the single-line lib.rs hunk. Tree anchor at dispatch: 6f193f90c6533329877804577c02d75ca57b2f98.”)
@yah:handoff(“REMAINING WORK IS FILED, NOT PARKED HERE: R861-T2 (blocked_on operator) carries the one open call — the fleet is still on policy.mode: file, so the reconciler WRITES acls.yaml rather than making it stop existing. litestream replicates only headscale.db, so policy is now reconstructible-from-declared-config but is still unreplicated on-disk state. Flipping to mode=database closes it completely; that is a live-infrastructure change with a migration sequence and no defensible default, so it was deliberately left alone.”)
Structs§
- Headscale
Client - HTTP client for a running Headscale coordinator.
- Headscale
Health - Headscale version + health response.
- Node
Info - Summary information about a node in the Headscale tailnet.
- Preauth
Key - A Headscale pre-authentication key for onboarding a new node.
- Preauth
KeyRecord - One pre-auth key as the coordinator reports it (
GET /api/v1/preauthkey). - Preauth
KeyRequest - Everything
POST /api/v1/preauthkeyneeds. Split out fromHeadscaleClient::create_preauth_key’s fixed single-use/1-hour shape so a reconciler can mint the standing, reusable keys a declared spec asks for.
Constants§
- DEFAULT_
ACL_ POLICY - A permissive ACL policy that allows all nodes to communicate.
Can be refined later with
yah mesh acl edit. - HEADSCALE_
VERSION - Pinned Headscale release version used by
yah mesh start. - POLICY_
MODE - The
policy.modeevery in-tree renderer emits (R861-T2).
Functions§
- cloudflare_
credentials - Load Cloudflare credentials from vault or environment.
- generate_
headscale_ config - Generate a headscale YAML configuration for Phase 1a (camp-local coordinator).
- headscale_
download_ url - Return the GitHub release download URL for headscale on this platform.
- update_
cloudflare_ dns - Update (or create) the A record for
record_namein the given Cloudflare zone, pointing it atnew_ip. Credentials come from the caller — usecloudflare_credentialsto load them from the vault / env.