zc2 0.0.43

P2P compute broker with credit-based billing, WAL, and broker mesh support
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
![zakuro Logo](https://github.com/zakuro-ai/zc/raw/HEAD/imgs/zakuro-banner.png)

<div align="center">
  <h1>zc</h1>
  <p>P2P compute broker with credit-based billing, automatic worker discovery, and real-time monitoring.</p>
</div>

<p align="center">
  <a href="#overview">Overview</a> •
  <a href="#installation">Installation</a> •
  <a href="#macos-menu-bar-app-and-widget">macOS App</a> •
  <a href="#quick-start">Quick Start</a> •
  <a href="#zakuro-drive-hooks">Drive Hooks</a> •
  <a href="#broker">Broker</a> •
  <a href="#benchmarking">Benchmarking</a> •
  <a href="#api-endpoints">API Endpoints</a> •
  <a href="#development">Development</a>
</p>

## Overview

`zc` is the Zakuro AI command-line tool that acts as a P2P compute broker. It routes compute requests to workers across a distributed network, handles credit-based billing via the Zakuro dashboard API with write-ahead logging, automatically discovers workers on the WireGuard mesh, and reports live transactions, workers and metrics from the CLI and the broker's `/stats` endpoint.

**Key features:**

- **P2P compute routing** — forward requests to optimal workers based on price, latency, or availability
- **P2P broker mesh** — broker-to-broker communication with authoritative user assignment (ZAKURO_P2P mode)
- **QUIC transport** — high-throughput multiplexed UDP transport for peer task offers (QUIC-first with HTTP fallback)
- **6 routing strategies** — `best_price`, `best_latency`, `best_availability`, `round_robin`, `random`, `weighted_capacity`
- **Credit-based billing** — dashboard API-backed ledger with atomic reserve-commit transactions; no direct database access; standalone mode (`ZAKURO_MASTER_KEY`) runs a local ledger without dashboard
- **Write-ahead log** — JSONL WAL with DashMap in-memory index, batched fsync, and crash recovery
- **Zero-latency hot path** — in-memory DashMap credit operations for authoritative users (P2P mode)
- **Automatic discovery** — finds workers on the mesh subnet (parallel scan) or falls back to localhost
- **Built-in benchmarking** — measure throughput/latency and compare routing strategies
- **Cluster management** — Docker container lifecycle, image management, and diagnostics
- **Zakuro Drive hooks** — manage file-event hooks (webhook/exec) on a local [zakuro-drive](https://github.com/zakuro-ai/zakuro-drive) daemon over its QUIC API

## Installation

### Prerequisites

- [Rust](https://rustup.rs/) — recent stable (the dependency tree's MSRV is 1.88 via `time` 0.3.47; the old "1.75+" note predates that)
- [Task](https://taskfile.dev/) (replaces Make)
- Docker and Docker Compose (optional, for containerized deployments)
- A Zakuro API key (`ZAKURO_API_KEY`) — sign up at [zakuro-ai.com](https://zakuro-ai.com)

### Prebuilt binary

```bash
curl -fsSL https://get.zakuro-ai.com/zc | bash   # install
zc update                                        # update in place
```

`get.zakuro-ai.com` is the canonical binary source: a verified mirror of the
[GitHub releases](https://github.com/zakuro-ai/zc/releases). Both the installer
and `zc update` read it, so they give the same version, and neither needs a
GitHub token. `zc update` checks the download's sha256 against the manifest
before replacing the binary and refuses on a mismatch. If the mirror is down and
a GitHub token is available (`gh auth login`, `GH_PAT` or `GITHUB_TOKEN`), it
falls back to the GitHub releases. See [ZC-0015](docs/adr/0015-distribution-paths.md).

### From Source

```bash
git clone https://github.com/zakuro-ai/zc.git
cd zc
task build:bin
```

This compiles the release binary and installs it to `/usr/local/bin/zc`.

#### Install with Cargo (no clone)

```bash
cargo install --git https://github.com/zakuro-ai/zc zc2
```

`zc2` needs **no access to any private repository** — it simply does not
include `zc hooks` (Zakuro Drive file-event hooks). That command lives in a
separate `zc-hooks` helper binary which `zc hooks` execs, because it depends on
the private `zakuro-client` crate. Building it requires read access to
`zakuro-ai/zakuro-drive`:

```bash
cargo build --release -p zc-hooks
```

Put the resulting `zc-hooks` next to your `zc` binary (or anywhere on `PATH`)
and `zc hooks …` works as before.

### From crates.io

```bash
cargo install zc2
```

Add `--locked` to get the exact dependency tree CI tests; without it cargo
resolves the newest compatible versions.

Note the registry build has no `zc hooks` — that command needs the `zc-hooks`
sidecar, which depends on a private crate and so cannot be published. Use a
[binary release](https://github.com/zakuro-ai/zc/releases) if you need it.

### With Docker

```bash
task build
```

Builds the binary, then builds and starts the Docker services via `docker compose`.

### macOS menu bar app and widget

On a Mac, `Zakuro.app` puts the same box in your menu bar and on your desktop: sharing state,
credit balance, earnings, devices online, and — from the medium widget — Pause/Resume and price
controls that write straight to `zc agent`.

```bash
export ZAKURO_DEVELOPMENT_TEAM=<your Team ID>
task macos:install      # build signed, install to /Applications, relaunch
zc login && zc agent install
# open the popover once, then: right-click the desktop -> Edit Widgets... -> "Zakuro"
```

Requires macOS 14+, Xcode 26 and `xcodegen`. A zc release ships the CLI binary only, so the app
is built from source. Opening the popover once is required before the widget shows anything —
it is what hands the sandboxed widget the agent's address and token.

Full instructions, including the Team ID lookup and troubleshooting: **[`macos/README.md`](macos/README.md)**.

### Verify

```bash
zc --version
```

## Quick Start

```bash
# 1. Set your API key
export ZAKURO_API_KEY="zk_myuser_abc123"

# 2. Set dashboard API URL
export ZAKURO_API_URL="https://hub.zakuro-ai.com"

# 3. Start the broker
zc broker

# The broker discovers workers, listens on 0.0.0.0:9000,
# and routes all credit operations through the dashboard API.
```

## Zakuro Drive Hooks

`zc hooks` manages file-event hooks on a running [zakuro-drive](https://github.com/zakuro-ai/zakuro-drive) daemon (`zakurod`) over its QUIC API. A hook fires on file events (`file.created` | `file.modified` | `file.deleted`) matching a glob, and either POSTs a signed JSON payload to a webhook or runs a local command.

```bash
# Webhook hook — secret read from stdin (recommended; avoids ps/history leaks)
echo "$WEBHOOK_SECRET" | zc hooks add --name slack-notify \
  --on file.created --on file.modified --glob "reports/**/*.pdf" \
  --webhook https://hooks.example.com/zakuro --secret-stdin

# Local command hook with a custom timeout
zc hooks add --name csv-to-parquet --on file.created --glob "**/*.csv" \
  --exec "/usr/local/bin/convert-csv" --timeout 120

zc hooks list --all          # include disabled hooks
zc hooks get csv-to-parquet  # full config (secrets redacted by the daemon)
zc hooks update csv-to-parquet --disable
zc hooks logs --failed --from 2h
zc hooks remove csv-to-parquet
```

The daemon address defaults to `127.0.0.1:4719`; override with `--daemon HOST:PORT` or `ZAKURO_DRIVE_DAEMON`. Run `zc hooks --help` for the full flag reference (retry tuning, `--secret-from-env`, `--tls-insecure`, dead-letter views).

> **Packaging note:** `zakuro-client` (the drive daemon API client) is consumed as a git dependency on the private `zakuro-ai/zakuro-drive` repository, pinned to the `client-v0.1.0` tag. crates.io rejects git dependencies, so the crates.io publish job in `auto-release.yml` is paused until `zakuro-proto` + `zakuro-client` are published to a registry; GitHub releases with prebuilt binaries continue.

## Broker

The broker accepts compute requests, selects an optimal worker, forwards the request, and handles billing.

### Modes

| Mode | Command | Description |
|------|---------|-------------|
| Foreground | `zc broker` | Live transaction log with colored output |
| Daemon | `zc -d broker` | Background process, minimal output |

```bash
zc broker                # foreground on :9000
zc -d broker             # daemon on :9000
zc broker 8080           # custom port
zc broker 0.0.0.0 9000   # custom host and port
```

### Worker Discovery

| Mode | Trigger | Behavior |
|------|---------|----------|
| **Mesh** | a `zakuro0` or `wg*` interface with a `10.13.13.x` address, or `ZAKURO_WIREGUARD_IP` env | Scans peers + subnet; credits enforced for remote workers |
| **Peers** | `ZAKURO_PEERS` env set (no mesh interface) | Probes explicit peer addresses; credits enforced |
| **Local** | No mesh, no peers | Scans `localhost:8089-8091`; execution is free |

Workers must expose `/health` and `/info` endpoints to be discovered. Workers can also [register manually](#post-workers) via the API.

In a **P2P mesh**, each broker detects its own mesh IP and treats workers at `localhost`/`127.0.0.1` or at that IP as **local** (free execution). Remote workers (mesh peers) are charged via the credit ledger. See [docs/USAGE.md](docs/USAGE.md#p2p-mesh-deployment) for mesh setup instructions.

Subnet scanning is **parallelized**: phase 1 does a concurrent TCP SYN to all 253 /24 hosts (50ms timeout); phase 2 probes only reachable hosts. This reduces scan time from `253 × 50ms` to `~50ms + probe time`.

### P2P Broker-to-Broker Communication

Enable distributed broker mesh with `ZAKURO_P2P=true`. Each broker becomes authoritative for a subset of users:

| Mode | Description |
|------|-------------|
| **Authority::Local** | User owned by this broker (via FNV-1a hash) → in-memory DashMap, zero PG on hot path |
| **Authority::Peer(url)** | User owned by remote broker → HTTP `/peer/reserve` + `/peer/commit` (~1ms RTT) |
| **Authority::Standalone** | Peer unreachable → dashboard API fallback path |

**Peer endpoints** (`X-Peer-Key` auth):
- `POST /peer/reserve` — reserve credits for remote user
- `POST /peer/commit` — commit transaction after execution
- `POST /peer/cancel` — cancel reservation (refund)
- `GET /peer/balance` — get balance for remote user
- `GET /peer/health` — peer health check

**Environment variables:**
- `ZAKURO_P2P=true` — enable P2P mode (default: false)
- `ZAKURO_PEER_KEY` — shared secret for peer authentication
- `ZAKURO_OWNER_ID` — this broker's ID for authority assignment
- `ZAKURO_PEERS` — comma-separated peer broker URLs for discovery

### Routing Strategies

| Strategy | Description |
|----------|-------------|
| `best_price` | Cheapest worker **(default)** |
| `best_latency` | Fastest response time |
| `best_availability` | Most available resources |
| `round_robin` | Even distribution |
| `random` | Random selection |
| `weighted_capacity` | Weighted by available CPU/memory |

Set via the `strategy` field in `X-Zakuro-Requirements` header on `/execute` requests.

### Credit System

- **Dashboard API ledger** — all balance and transaction operations go through `ZAKURO_API_URL`; no direct database access from the broker. The dashboard returns balance in the `credits_balance` field.
- **Standalone mode** — set `ZAKURO_MASTER_KEY` (no `ZAKURO_API_URL` needed) to run a fully local credit ledger. Transactions are queued in the WAL and the flush buffer; billing is enforced even without the dashboard.
- **P2P mode** (`ZAKURO_P2P=true`) — authoritative broker owns users via FNV-1a hash, DashMap in-memory operations on the hot path. Setting `remote_only: true` in a request forces routing to peer workers (bypasses local worker selection).
- **Write-ahead log** — append-only JSONL with DashMap index (O(1) lookup), batched fsync (100ms/64 entries), crash recovery, auto-compaction every ~5 min
- **Flush thread** — batched writes every 5s (transactions + balance snapshots), synced to the dashboard API
- **Local mode** — all execution is free when no mesh network is detected

```bash
# Check balance (requires Bearer token — own user or admin)
curl http://localhost:9000/credits/my-user \
  -H "Authorization: Bearer $API_KEY"

# Add credits (requires master API key)
curl -X POST http://localhost:9000/credits/my-user/add \
  -H "Content-Type: application/json" \
  -H "X-Api-Key: $ZAKURO_MASTER_KEY" \
  -d '{"amount": 100.0, "description": "Initial deposit"}'
```

## Model Inference (`zc infer`)

```bash
# Chat model: one-shot prompt
zc infer zc://<owner>/<name> -m "Say hello" [--max-tokens N] [--system "<sys>"] [--broker <url>]

# Speech model: transcribe an audio file (served through the hub)
zc infer zc://<owner>/<name> --file clip.wav [--language sn]
```

`-m` and `--file` are mutually exclusive. `--file` accepts wav, mp3, flac, m4a,
ogg and webm; `--language` is an optional 2-3 letter code (`^[a-z]{2,3}$`). The
transcript is printed first, then a receipt (language, duration, billed
seconds, cost in zkcr). Audio longer than 10 minutes is transcribed up to that
limit and a warning says so. Hub errors (insufficient credits, model not
committed, a chat model given `--file`) are printed verbatim with ` (hub)`.
It uses the API URL and key saved by `zc login` (or `ZAKURO_API_URL` /
`ZAKURO_API_KEY`).

## Benchmarking

```bash
zc bench                          # default (10 concurrent, 1000 requests)
zc bench -c 100 -n 10000         # high-load
zc bench -s best_latency         # specific strategy
zc bench --compare               # compare all strategies
zc bench http://broker:9000      # remote broker
zc bench mesh --workers a:3960,b:3960,c:3960 | jq .
```

| Option | Short | Default | Description |
|--------|-------|---------|-------------|
| `--concurrency` | `-c` | `10` | Concurrent workers |
| `--requests` | `-n` | `1000` | Total requests |
| `--url` | `-u` | `http://127.0.0.1:9000` | Broker URL |
| `--strategy` | `-s` | `best_price` | Routing strategy |
| `--compare` | | | Compare all strategies |
| `--list-strategies` | | | List strategies |

Output includes throughput (RPS), latency percentiles (p50-p99), success/failure rates, and an ASCII histogram.

`zc bench mesh` emits a JSON mesh-warmup report with per-worker `latency_p95`,
`bandwidth_bps`, and `recommended_backpressure`, suitable for `jq` or for
feeding into operator tooling.

## Remote Monitoring

Read a running broker from another machine:

```bash
zc workers localhost:9000            # its workers, their health and prices
zc workers http://broker.mesh:9000
# transactions, request rate and latency percentiles, as JSON
curl -s -H "Authorization: Bearer $ZAKURO_API_KEY" http://broker.mesh:9000/stats
```

## System Diagnostics

```bash
zc info
```

Shows network info (hostname, mode, IPs), compute cluster availability (Ray, Dask, Spark), environment variables, and broker port status.

## API Endpoints

The broker listens on `0.0.0.0:9000` by default.

| Method | Endpoint | Auth | Description |
|--------|----------|------|-------------|
| GET | `/health` | None | Health check |
| GET | `/stats` | Bearer | Broker statistics (non-admin: own txns only) |
| GET | `/workers` | None | List registered workers |
| POST | `/workers` | `X-Worker-Key` | Register a worker |
| POST | `/workers/heartbeat` | `X-Worker-Key` | Worker heartbeat |
| DELETE | `/workers/:id` | `X-Worker-Key` | Unregister worker |
| POST | `/execute` | Bearer | Execute compute request |
| POST | `/price` | None | Estimate request price |
| POST | `/route` | None | Placement dry run: same filtering + strategy ranking as `/execute`, executes/reserves/bills/mutates nothing |
| GET | `/credits/:user` | Bearer | Get balance (own user or admin) |
| POST | `/credits/:user/add` | `X-Api-Key` (master) | Add credits |
| GET | `/me` | Bearer | Current user info |
| GET | `/ledger/status` | Bearer (admin) | Ledger status (API URL, connection state) |
| **P2P Endpoints** (ZAKURO_P2P=true) | | | |
| POST | `/peer/reserve` | `X-Peer-Key` | Reserve credits for remote user |
| POST | `/peer/commit` | `X-Peer-Key` | Commit transaction after execution |
| POST | `/peer/cancel` | `X-Peer-Key` | Cancel reservation (refund) |
| POST | `/peer/earn` | `X-Peer-Key` | Credit earnings to broker owner |
| GET | `/peer/balance` | `X-Peer-Key` | Get balance for remote user |
| GET | `/peer/workers` | `X-Peer-Key` | List this broker's local workers |
| GET | `/peer/health` | `X-Peer-Key` | Peer broker health check |

### Execute Request Options

The `/execute` endpoint accepts a JSON body (or `X-Zakuro-Requirements` header) with:

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `cpus` | float | `1.0` | Required vCPUs |
| `memory_bytes` | int | — | Required memory in bytes |
| `gpus` | int | `0` | Required GPUs |
| `estimated_duration_secs` | float | `1.0` | Estimated job duration (used for cost estimation) |
| `budget_credits` | float | — | Maximum credits to spend; request fails if cost would exceed this. Must be positive when set. **Required for remote dispatch when vouchers are on** (`ZAKURO_REQUIRE_VOUCHER`, the default for a CLI broker with billing credentials): the broker mints the job's voucher for this amount, and a job sent to a peer without it fails with `400 BUDGET_REQUIRED`. There is no default yet; it will come from the `policy.toml` `[budget]` (ZC-0017) |
| `strategy` | string | `best_price` | Routing strategy |
| `remote_only` | bool | `false` | When `true`, skip local workers and dispatch only to peer brokers (requires `ZAKURO_P2P=true`) |
| `worker_type` | string | — | Filter workers by type (e.g. `standard`, `premium`) |
| `min_vram_gb` | int | — | Minimum advertised GPU VRAM (GiB) a candidate must have; a worker that advertises none is excluded once this is set |
| `tags` | string[] | — | Capability tags every candidate must carry (e.g. `["apple-silicon"]`). No healthy worker with all of them -> `503 NO_WORKERS_WITH_TAGS` naming the missing tag; never a fallback to an untagged node |

**Capability tags.** A broker stamps the workers it owns with `apple-silicon`
when it runs on macOS/aarch64 (detected at runtime), plus any operator tags in
`ZAKURO_NODE_TAGS=a,b`; a worker's own `/info` tags are kept. Tags travel on
`/workers` and `/peer/workers` (`tags`, omitted when empty, ignored by older
peers), so a tagged worker on another node is placeable. Example (MLX export
step, sakura-internal#66):
`-H 'X-Zakuro-Requirements: {"tags":["apple-silicon"],"cpus":2}'`.

Response headers include `X-Zakuro-Worker` (worker ID), `X-Zakuro-Cost` (credits spent), and `X-Zakuro-Credits-Remaining` (balance after execution).

### Placement Dry Run (`POST /route`)

Preview what `/execute` would do right now, for the same `ResourceRequirements`
JSON body `/execute` accepts (`cpus`, `memory_bytes`, `gpus`, `min_vram_gb`,
`strategy`, `worker_type`, `tags`, ...) -- without executing, reserving,
billing, or mutating any broker state (no quota reservation, no round-robin
cursor advance, no in-flight counters touched). Runs the exact router code
path `/execute` uses: same eligibility filtering, same strategy ranking.

```bash
curl -s -X POST http://localhost:9000/route \
  -H 'Content-Type: application/json' \
  -d '{"strategy": "best_price", "gpus": 1, "min_vram_gb": 16}' | jq
```

Response (200):

```json
{
  "strategy": "best_price",
  "chosen": {
    "worker_id": "...", "node_fingerprint": "...", "address": "zc://worker-...",
    "gpu_model": "RTX 4090", "gpu_vram_gb": 24, "price_per_hour": 1.8,
    "estimated_cost": 0.42, "active_requests": 0,
    "atelier_runner": "http://10.13.13.9:8700"
  },
  "candidates": [ /* same shape, ranked best first; chosen == candidates[0] */ ],
  "excluded": [ { "worker_id": "...", "reason": "gpu_vram_gb 11 < min_vram_gb 16" } ]
}
```

A worker advertises an atelier runner endpoint via the tag
`atelier-runner=<http url>`; when present it is surfaced as `atelier_runner`
on `chosen` and on each candidate. When nothing is eligible, the response is
404 with `{"error": ..., "code": "NO_CAPACITY", "excluded": [...]}` listing
why every healthy worker was ruled out.

`chosen` is the policy's first eligible choice, not a dispatch guarantee:
`/execute` retries past a first choice that turns out to be at quota and
tries the rest before giving up. `/route` has no request to reserve against,
so it cannot observe or reproduce that -- `chosen` can name a worker
`/execute` ends up skipping at the moment it actually runs.

### Worker Registration

```json
{
  "name": "worker-1",
  "uri": "http://10.13.13.5:8089",
  "worker_type": "zakuro",
  "resources": {
    "cpus_total": 8.0,
    "cpus_available": 8.0,
    "memory_total": 17179869184,
    "memory_available": 17179869184,
    "gpus_total": 1,
    "gpus_available": 1
  },
  "pricing": {
    "cpu_price": 0.001,
    "memory_price": 0.0001,
    "gpu_price": 0.01,
    "min_charge": 0.001
  }
}
```

## Cluster Commands

| Command | Description |
|---------|-------------|
| `zc up` | Start 1 worker + broker locally (foreground) |
| `zc up --workers N` | Start N workers + broker |
| `zc down` | Stop all workers + broker started by `zc up` |
| `zc workers` | List workers registered with the local broker |
| `zc update` | Update CLI from get.zakuro-ai.com |
| `zc pull` | Pull updated Docker images |
| `zc images` | List Zakuro images |
| `zc ps` | List running containers |
| `zc kill` | Stop running containers |
| `zc restart` | Restart containers |

## Environment Variables

| Variable | Required | Default | Description |
|----------|----------|---------|-------------|
| `ZAKURO_API_KEY` | No* | — | Authentication token for dashboard API. *Optional when using only local or discovered brokers (P2P only). |
| `ZAKURO_MASTER_KEY` | No | — | Master API key for admin operations and standalone billing. When set (without `ZAKURO_API_URL`), the broker runs a fully local credit ledger with billing enforced. |
| `ZAKURO_API_URL` | No* | — | Dashboard API URL (e.g., `https://hub.zakuro-ai.com`). *Required for dashboard-backed billing and worker sync. Omit to use standalone mode (`ZAKURO_MASTER_KEY`). |
| `ZAKURO_WORKER_KEY` | No | — | Shared secret for worker management endpoints (`POST/DELETE /workers`, heartbeat) |
| `ZAKURO_WIREGUARD_IP` | No | — | Override mesh IP detection (Docker sidecar setups). Named for the mesh we used to run; it applies to WireGuard too. |
| `ZAKURO_PEERS` | No | — | Comma-separated `ip:port` peers for explicit discovery (workers) or `http://host:port` (brokers in P2P mode) |
| `ZAKURO_DISCOVER_BROKER_PEERS` | No | `true` | When P2P is enabled and `ZAKURO_PEERS` is empty, scan localhost ports 9000–9010 for other brokers. Set to `false` to disable. When using discovered brokers, `ZAKURO_API_KEY` is optional. |
| `ZAKURO_P2P` | No | `false` | Enable P2P broker-to-broker communication |
| `ZAKURO_PEER_KEY` | No | — | Shared secret for P2P broker authentication (`X-Peer-Key` header) |
| `ZAKURO_OWNER_ID` | — | — | **Not configurable.** Always derived from `ZAKURO_API_KEY` (`zk_{user_id}_{hex}` format). Manual overrides are ignored. |
| `ZAKURO_NODE_NAME` | No | Auto-detected | Human-readable node name (falls back to the mesh hostname, then system hostname) |
| `ZAKURO_SHARE_RUNTIME` | No | `auto` | Where `zc share` / `zc up` run the worker: `auto` (the `zakuro-image-compute` container when Docker is reachable, else the host `uv run` process with a one-line warning), `host` (always the host process) or `container` (require Docker; fails clearly without it). The container publishes its port on `127.0.0.1` only. `ZC_SHARE_WORKER_IMAGE` overrides the image (use `name@sha256:...` to pin). |
| `ZAKURO_WORKER_PORT` | No | `3960` | Primary port workers listen on (discovery) |
| `ZAKURO_SCAN_RANGE` | No | — | Port range for discovery scan, e.g. `3960-3999` |
| `ZAKURO_SCAN_INTERVAL` | No | `15` | Seconds between discovery scans |
| `ZAKURO_AUTO_UPDATE` | No | off | Opt-in broker self-update from crates.io (crate `zc2`). `1`/`true`/`yes`/`on` enables it; anything else is off. Never set it on k8s fleets (their image is managed by CD). When on, the broker reads the crates.io sparse index; for a strictly newer non-yanked release it runs `cargo install zc2 --version X --locked` into a scratch dir next to the executable while it keeps serving, checks the built binary runs and reports exactly X, waits until idle (no in-flight requests, local workers or pending settlements, on 3 polls 5 s apart), swaps it in atomically and exits with code 75 so systemd (`Restart=always`) starts it. Needs `cargo` (see `ZAKURO_CARGO`) and a writable executable directory; failures are logged and retried next interval. Status (`source`, `building`, `last_error`, ...) is in `GET /health` under `auto_update`. `zc update` still uses the release mirror. |
| `ZAKURO_AUTO_UPDATE_INTERVAL` | No | `600` | Time between index checks, in seconds or with an `s`/`m`/`h` suffix (`10m`, `1h`). Minimum 300 s. +up to 10% jitter; the first check is after a random 60 s..interval delay. |
| `ZAKURO_AUTO_UPDATE_INDEX` | No | `https://index.crates.io/3/z/zc2` | Sparse-index URL for `zc2` (https; http only for `127.0.0.1`/`localhost`). For tests. |
| `ZAKURO_CARGO` | No | `cargo` on PATH, else `$HOME/.cargo/bin/cargo` | The cargo used by auto-update builds. |
| `ZAKURO_ENV` | No | `prod` | Environment profile: `lab`, `staging` (`stg`, `stage`) or `prod`. Sets the hub API, the get/update mirror, the registry and the trust roots together (`zc doctor` prints them). `lab` = `http://hub.lab.zakuro.internal`, `http://get.lab.zakuro.internal`, `registry.lab.zakuro.internal:5000`, and skips the legacy `/env` fetch into `/opt/zakuro/etc/zk0.env`. Also read from `env=` in `~/.zakuro/credentials` or `ZAKURO_ENV=` in `~/.zakuro/env`. `ZAKURO_API_URL`, `ZAKURO_UPDATE_URL` and `ZAKURO_REGISTRY` still override their part. |
| `ZAKURO_REGISTRY` | No | the profile's | OCI registry override for the active `ZAKURO_ENV` profile. |
| `ZAKURO_UPDATE_URL` | No | the profile's (`https://get.zakuro-ai.com/zc`) | Mirror base for `zc update` (the manual path; broker auto-update uses crates.io). Either a bare host base (`https://stg.get.zakuro-ai.com`) or one ending in `/zc`; trailing `/` is trimmed and `/zc` appended when missing. Must be `https://` (plain `http://` only for `127.0.0.1`/`localhost`); an invalid value is an error, never silently ignored. |

## Development

### Task Commands

This project uses [Task](https://taskfile.dev/) as its task runner. Install it with `go install github.com/go-task/task/v3/cmd/task@latest` or see [installation docs](https://taskfile.dev/installation/).

| Command | Description |
|---------|-------------|
| `task build` | Build binary and start Docker services |
| `task build:bin` | Incremental release build of `zc` and `zc-hooks`, installed to `~/.local/bin` |
| `task dev` | Incremental debug build for fixing bugs, linked as `~/.local/bin/zc-dev` |
| `task build:docker` | Build Docker image only |
| `task run` | Build and run the CLI |
| `task debug` | Build and run with `RUST_BACKTRACE=full` |
| `task release` | Run the installed release binary |
| `task broker` | Start broker in foreground |
| `task broker:daemon` | Start broker in daemon mode |
| `task bench` | Run default benchmark |
| `task bench:compare` | Compare all routing strategies |
| `task info` | Show system diagnostics |
| `task docker:up` | Start Docker services |
| `task docker:down` | Stop Docker services |
| `task docker:logs` | Tail Docker service logs |
| `task clean` | Remove build artifacts |

### Docker

```bash
# Build and run with Docker Compose
task build

# Or build the image directly
docker build -t zakuro-ai/broker -f docker/Dockerfile .
docker run -p 9000:9000 -e ZAKURO_API_KEY=demo zakuro-ai/broker
```

### Project Structure

```
src/
  main.rs              # CLI entry point and command routing
  broker/
    mod.rs             # Broker state, config, and per-worker local detection
    server.rs          # HTTP server (tiny_http) with execute handler
    worker.rs          # Worker registry, resources, and health tracking
    router.rs          # Routing strategies and worker selection
    credits.rs         # Lightweight per-user balance cache and rate-limit tracker (reservation/history owned by Ledger)
    ledger.rs          # Central ledger: API mode (dashboard) or local in-memory reserve-commit
    peer.rs            # P2P broker-to-broker communication (PeerClient, PeerManager, FNV-1a authority)
    flush.rs           # TransactionBuffer for batched writes (5s cycle, via dashboard API)
    discovery.rs       # mesh/local/peer-based worker discovery
    wal.rs             # Write-ahead log (JSONL + DashMap index, batched fsync)
    recovery.rs        # WAL replay on startup (reserved→cancel, executed→commit)
    bench.rs           # Benchmarking tool
    stats.rs           # Statistics tracking
    info.rs            # System diagnostics
  common/mod.rs        # Shared utilities
  exec/mod.rs          # Command execution helpers
  manager/mod.rs       # Docker container management
  network/mod.rs       # Network interface detection
  envs/mod.rs          # Environment variable handling
```

## Troubleshooting

- **"You are missing ZAKURO_API_KEY"** — set `ZAKURO_API_KEY` for dashboard auth. For standalone billing, set `ZAKURO_MASTER_KEY` instead. P2P-only setups (auto-discovered peers) don't require either.
- **No workers discovered** — ensure workers expose `/health` and `/info` endpoints on port 3960 (or `ZAKURO_WORKER_PORT`). Check with `zc info`.
- **Credit operations failing** — ensure both `ZAKURO_API_KEY` and `ZAKURO_API_URL` are set for dashboard mode. For standalone mode, set only `ZAKURO_MASTER_KEY`. Verify the dashboard returns the `credits_balance` field in `/api/broker/balance/{id}`.
- **Zero cost on remote executions** — ensure `is_billing_enabled()` returns true: either dashboard URL+key or `ZAKURO_MASTER_KEY` must be set. Local workers (`127.0.0.1`/`localhost`) are always free; remote mesh workers are billed.
- **Remote job fails with `400 BUDGET_REQUIRED`** — the broker requires vouchers (`ZAKURO_REQUIRE_VOUCHER`) and the request had no positive `budget_credits`. Send one, e.g. `-H 'X-Zakuro-Requirements: {"remote_only": true, "budget_credits": 1.0}'`.
- **`remote_only: true` returns 503** — ensure `ZAKURO_P2P=true` and at least one peer broker is reachable. Without peers, `remote_only` requests cannot be served.
- **P2P mode not working** — ensure `ZAKURO_P2P=true`, `ZAKURO_PEER_KEY` matches on all brokers. Use `ZAKURO_PEERS` for explicit URLs, or leave empty to auto-discover brokers on localhost (ports 9000–9010).
- **Mesh IP not detected** — the broker recognises a `zakuro0` or `wg*` interface holding an address in `10.13.13.0/24` (see `get_mesh_ip` in `src/broker/discovery.rs`). An interface outside that subnet is ignored. Set `ZAKURO_WIREGUARD_IP` to override.

## Documentation

See [docs/USAGE.md](docs/USAGE.md) for the complete usage guide covering all API details, configuration reference, and the credit system in depth.

See [docs/QUIC_TRANSPORT.md](docs/QUIC_TRANSPORT.md) for details on the QUIC-based P2P transport layer.

## License

MIT