# A four-node dev mesh with four owners
Four `zc` nodes on Docker, each belonging to a different person. Three are
other people's machines; the fourth is yours. Send a job to your node, watch it
land on someone else's hardware, and watch your balance fall while theirs
rises — against the real marketplace ledger, not a simulation of one.
| node1 | `alice@mesh.local` | 10.42.0.11 | 19001 | 1.80 | RTX 4090 | 25 |
| node2 | `bob@mesh.local` | 10.42.0.12 | 19002 | 3.60 | A100 40GB | 25 |
| node3 | `carol@mesh.local` | 10.42.0.13 | 19003 | 7.20 | H100 80GB | 25 |
| node4 | **yours** | 10.42.0.14 | 19004 | 2.50 | RTX 4090 | 500 |
The containers are identical. The only thing that makes node4 yours is the key
it carries — ownership here is an identity, not a kind of hardware.
## The two credentials
This is the part that is easy to get wrong, because both are API keys and they
mean opposite things.
| **Owner** | `ZAKURO_API_KEY` in the container, sent as `X-Broker-Api-Key` | who owns the machine, who gets **paid**, whose name the node appears under |
| **Requester** | `Authorization: Bearer zk_…` on `/execute` | whose credits get **debited** |
So node4 carrying your key means *you get paid* when others use it. You
*spending* happens when you present your key as a Bearer token — to any node.
Knocking on your own node's door is not what makes a job yours.
## Quick start
The mesh bills against the marketplace, so that has to be up first.
```bash
(cd ../zak-marketplace && docker compose up -d api) # billing authority
./mesh/provision.sh # 4 accounts -> mesh/.env
export GH_TOKEN=$(gh auth token) # see "The build needs a token"
docker compose -f mesh/compose.yaml up -d --build # first build compiles zc
./mesh/verify.sh # 5 checks, all silent failures
```
### The build needs a token
Not to run — only to build, and only to *fetch* sources.
`zc` declares the private `zakuro-drive` repo as an optional dependency behind
the `hooks` feature, and that feature is off by default, so nothing here
compiles or links it. But cargo resolves the dependency graph before it resolves
features, and resolving a git dependency means fetching it. Measured: `cargo
build --release --locked` in a clean container still runs `git fetch
https://github.com/zakuro-ai/zakuro-drive.git` and dies with git exit 128 when
it cannot authenticate. `--locked` does not avoid it.
So the build takes a GitHub token as a BuildKit secret, uses it in exactly one
`cargo fetch` layer, and every layer after that is `--offline`. The token is
never written into an image layer, and the builder stage is discarded — only the
compiled binary is copied out.
`export GH_TOKEN=$(gh auth token)` is enough if you use the `gh` CLI. Forgetting
it fails at the fetch layer with `exit code 128`, early and loudly, rather than
after the ten-minute dependency compile.
`provision.sh` creates the four accounts through the marketplace's own
`provision_user()` helper and writes their keys to `mesh/.env` (gitignored, mode
600). Re-running it **rotates** the keys — a `zk_` key is stored only as a
sha256 hash, so there is nothing to read back and reuse. Re-run it and `up -d`
together, never one without the other.
## The payoff
```bash
curl -s -X POST http://localhost:19004/execute \
-H "Authorization: Bearer $KEY" \
-H 'X-Zakuro-Requirements: {"cpus":1.0,"estimated_duration_secs":10.0,"remote_only":true}' \
-d '{}' -i | grep -i 'x-zakuro-transport\|^{'
```
The requirements go in a **header**, not the body. The body is the payload
forwarded to the worker, so `-d '{"remote_only":true}'` is not a syntax error —
it runs the job with defaults, locally and free, and looks like it worked.
`remote_only` forces the job off your own worker, so:
- `x-zakuro-transport: subscribed` — a peer took it, not you
- the reply names which worker ran it (`{"status":"ok","worker":"worker-alice"}`)
- your balance drops, that owner's rises, and both show up in the dashboard
Without `remote_only` the broker prefers your own worker at `127.0.0.1`, which
is free — that is the local/free vs remote/paid split, working correctly.
Check the money moved:
```bash
docker compose -f ../zak-marketplace/compose.yaml exec -T postgres \
psql -U marketplace -d dashboard -c \
"SELECT email, credits_balance FROM users WHERE email LIKE '%mesh.local' ORDER BY email;"
```
Verify by **balance movement**, not by the status code. `verify_broker_api_key`
checks that a key is valid but not that it matches the `zakuro_user_id` in the
request body, so a mis-wired key still answers `200` — it just bills the wrong
person.
## Why it is built this way
Each of these is here because getting it wrong fails *silently*.
**Worker and broker share a container.** The broker treats a worker on
`127.0.0.1` as local, and local is free; a worker on another node is billed.
Splitting them would make every worker remote and erase the distinction.
**Prices differ across nodes.** With one shared price the router has nothing to
choose between, and "which node won" carries no information.
**No `ZAKURO_MASTER_KEY`.** It is the standalone alternative to a dashboard, and
it resolves *every* caller to `admin` — it would collapse all four identities
into one while everything still appeared to work. `entrypoint.sh` refuses to
boot if it is set.
**`ZAKURO_PEERS` lists IPs, never compose DNS names.** A broker inserts *itself*
into the hash ring as `<own_ip>:<port>` but peers as whatever the list spelled.
Mixing the two gives each node a differently-sorted ring, so they disagree about
who owns whose credits.
**Peer discovery is off.** The ring is built once at startup and never mutated,
so with discovery on, membership depends on which peers happened to answer
during boot — nodes started at different moments disagree about `total_brokers`.
**The subnet is 10.42.0.0/24, not 10.13.13.0/24.** `vpn::is_mesh_ip()` matches
`10.13.13.x` exactly and routes those peers through a CONNECT sidecar that does
not exist here.
**Every layer after the fetch is `--offline --locked`.** A build that could
reach the network might quietly resolve something newer than `Cargo.lock` pins,
so the mesh would stop being the code you checked out.
**Ports are 19001-19004.** Clear of the marketplace stack's 8000, 5173 and
9000/9001. Both stacks must run at once, so a collision would be fatal.
**The worker reports `gpu_model`.** `zc`'s own worker sync leaves it null, which
is why the dashboard's GPU column reads `--` for real workers. Filling it means
that path is exercised rather than merely tolerated.
## When it comes up wrong
A broken mesh reports healthy. These are the failures worth knowing by shape:
| healthy, `/workers` shows 1 | `ZAKURO_PEER_KEY` differs between nodes — every probe 401s and no error is logged |
| everything free, no transactions | a node has no `ZAKURO_API_KEY`, so `is_billing_enabled()` is false |
| `402 INSUFFICIENT_CREDITS` | the billing path working correctly — top that account up |
| ring totals disagree | a node was started with a different `ZAKURO_PEERS`, or DNS names crept in |
| worker found but never registered | `/info` is missing `worker_type`; discovery declines it without complaint |
| build fails, `git ... exit status 128` | `GH_TOKEN` is unset — see "The build needs a token" |
| earnings on the wrong account | two nodes share a key — check `verify.sh` step 4 |
## Teardown
```bash
docker compose -f mesh/compose.yaml down -v # -v also drops the WALs
```
The four accounts survive in the marketplace database on purpose — their
balances are the record of what the mesh did. To remove them:
```sql
DELETE FROM transactions WHERE user_id IN (SELECT id FROM users WHERE email LIKE '%mesh.local');
DELETE FROM users WHERE email LIKE '%mesh.local';
```