
Overview
zc is the Zakuro AI command-line tool that acts as a P2P compute broker. It routes compute requests to workers across a distributed network, handles credit-based billing via the Zakuro dashboard API with write-ahead logging, automatically discovers workers on the WireGuard mesh, and reports live transactions, workers and metrics from the CLI and the broker's /stats endpoint.
Key features:
- P2P compute routing — forward requests to optimal workers based on price, latency, or availability
- P2P broker mesh — broker-to-broker communication with authoritative user assignment (ZAKURO_P2P mode)
- QUIC transport — high-throughput multiplexed UDP transport for peer task offers (QUIC-first with HTTP fallback)
- 6 routing strategies —
best_price,best_latency,best_availability,round_robin,random,weighted_capacity - Credit-based billing — dashboard API-backed ledger with atomic reserve-commit transactions; no direct database access; standalone mode (
ZAKURO_MASTER_KEY) runs a local ledger without dashboard - Write-ahead log — JSONL WAL with DashMap in-memory index, batched fsync, and crash recovery
- Zero-latency hot path — in-memory DashMap credit operations for authoritative users (P2P mode)
- Automatic discovery — finds workers on the mesh subnet (parallel scan) or falls back to localhost
- Built-in benchmarking — measure throughput/latency and compare routing strategies
- Cluster management — Docker container lifecycle, image management, and diagnostics
- Zakuro Drive hooks — manage file-event hooks (webhook/exec) on a local zakuro-drive daemon over its QUIC API
Installation
Prerequisites
- Rust — recent stable (the dependency tree's MSRV is 1.88 via
time0.3.47; the old "1.75+" note predates that) - Task (replaces Make)
- Docker and Docker Compose (optional, for containerized deployments)
- A Zakuro API key (
ZAKURO_API_KEY) — sign up at zakuro-ai.com
Prebuilt binary
|
get.zakuro-ai.com is the canonical binary source: a verified mirror of the
GitHub releases. Both the installer
and zc update read it, so they give the same version, and neither needs a
GitHub token. zc update checks the download's sha256 against the manifest
before replacing the binary and refuses on a mismatch. If the mirror is down and
a GitHub token is available (gh auth login, GH_PAT or GITHUB_TOKEN), it
falls back to the GitHub releases. See ZC-0015.
From Source
This compiles the release binary and installs it to /usr/local/bin/zc.
Install with Cargo (no clone)
zc2 needs no access to any private repository — it simply does not
include zc hooks (Zakuro Drive file-event hooks). That command lives in a
separate zc-hooks helper binary which zc hooks execs, because it depends on
the private zakuro-client crate. Building it requires read access to
zakuro-ai/zakuro-drive:
Put the resulting zc-hooks next to your zc binary (or anywhere on PATH)
and zc hooks … works as before.
From crates.io
Add --locked to get the exact dependency tree CI tests; without it cargo
resolves the newest compatible versions.
Note the registry build has no zc hooks — that command needs the zc-hooks
sidecar, which depends on a private crate and so cannot be published. Use a
binary release if you need it.
With Docker
Builds the binary, then builds and starts the Docker services via docker compose.
macOS menu bar app and widget
On a Mac, Zakuro.app puts the same box in your menu bar and on your desktop: sharing state,
credit balance, earnings, devices online, and — from the medium widget — Pause/Resume and price
controls that write straight to zc agent.
&&
# open the popover once, then: right-click the desktop -> Edit Widgets... -> "Zakuro"
Requires macOS 14+, Xcode 26 and xcodegen. A zc release ships the CLI binary only, so the app
is built from source. Opening the popover once is required before the widget shows anything —
it is what hands the sandboxed widget the agent's address and token.
Full instructions, including the Team ID lookup and troubleshooting: macos/README.md.
Verify
Quick Start
# 1. Set your API key
# 2. Set dashboard API URL
# 3. Start the broker
# The broker discovers workers, listens on 0.0.0.0:9000,
# and routes all credit operations through the dashboard API.
Zakuro Drive Hooks
zc hooks manages file-event hooks on a running zakuro-drive daemon (zakurod) over its QUIC API. A hook fires on file events (file.created | file.modified | file.deleted) matching a glob, and either POSTs a signed JSON payload to a webhook or runs a local command.
# Webhook hook — secret read from stdin (recommended; avoids ps/history leaks)
|
# Local command hook with a custom timeout
The daemon address defaults to 127.0.0.1:4719; override with --daemon HOST:PORT or ZAKURO_DRIVE_DAEMON. Run zc hooks --help for the full flag reference (retry tuning, --secret-from-env, --tls-insecure, dead-letter views).
Packaging note:
zakuro-client(the drive daemon API client) is consumed as a git dependency on the privatezakuro-ai/zakuro-driverepository, pinned to theclient-v0.1.0tag. crates.io rejects git dependencies, so the crates.io publish job inauto-release.ymlis paused untilzakuro-proto+zakuro-clientare published to a registry; GitHub releases with prebuilt binaries continue.
Broker
The broker accepts compute requests, selects an optimal worker, forwards the request, and handles billing.
Modes
| Mode | Command | Description |
|---|---|---|
| Foreground | zc broker |
Live transaction log with colored output |
| Daemon | zc -d broker |
Background process, minimal output |
Worker Discovery
| Mode | Trigger | Behavior |
|---|---|---|
| Mesh | a zakuro0 or wg* interface with a 10.13.13.x address, or ZAKURO_WIREGUARD_IP env |
Scans peers + subnet; credits enforced for remote workers |
| Peers | ZAKURO_PEERS env set (no mesh interface) |
Probes explicit peer addresses; credits enforced |
| Local | No mesh, no peers | Scans localhost:8089-8091; execution is free |
Workers must expose /health and /info endpoints to be discovered. Workers can also register manually via the API.
In a P2P mesh, each broker detects its own mesh IP and treats workers at localhost/127.0.0.1 or at that IP as local (free execution). Remote workers (mesh peers) are charged via the credit ledger. See docs/USAGE.md for mesh setup instructions.
Subnet scanning is parallelized: phase 1 does a concurrent TCP SYN to all 253 /24 hosts (50ms timeout); phase 2 probes only reachable hosts. This reduces scan time from 253 × 50ms to ~50ms + probe time.
P2P Broker-to-Broker Communication
Enable distributed broker mesh with ZAKURO_P2P=true. Each broker becomes authoritative for a subset of users:
| Mode | Description |
|---|---|
| Authority::Local | User owned by this broker (via FNV-1a hash) → in-memory DashMap, zero PG on hot path |
| Authority::Peer(url) | User owned by remote broker → HTTP /peer/reserve + /peer/commit (~1ms RTT) |
| Authority::Standalone | Peer unreachable → dashboard API fallback path |
Peer endpoints (X-Peer-Key auth):
POST /peer/reserve— reserve credits for remote userPOST /peer/commit— commit transaction after executionPOST /peer/cancel— cancel reservation (refund)GET /peer/balance— get balance for remote userGET /peer/health— peer health check
Environment variables:
ZAKURO_P2P=true— enable P2P mode (default: false)ZAKURO_PEER_KEY— shared secret for peer authenticationZAKURO_OWNER_ID— this broker's ID for authority assignmentZAKURO_PEERS— comma-separated peer broker URLs for discovery
Routing Strategies
| Strategy | Description |
|---|---|
best_price |
Cheapest worker (default) |
best_latency |
Fastest response time |
best_availability |
Most available resources |
round_robin |
Even distribution |
random |
Random selection |
weighted_capacity |
Weighted by available CPU/memory |
Set via the strategy field in X-Zakuro-Requirements header on /execute requests.
Credit System
- Dashboard API ledger — all balance and transaction operations go through
ZAKURO_API_URL; no direct database access from the broker. The dashboard returns balance in thecredits_balancefield. - Standalone mode — set
ZAKURO_MASTER_KEY(noZAKURO_API_URLneeded) to run a fully local credit ledger. Transactions are queued in the WAL and the flush buffer; billing is enforced even without the dashboard. - P2P mode (
ZAKURO_P2P=true) — authoritative broker owns users via FNV-1a hash, DashMap in-memory operations on the hot path. Settingremote_only: truein a request forces routing to peer workers (bypasses local worker selection). - Write-ahead log — append-only JSONL with DashMap index (O(1) lookup), batched fsync (100ms/64 entries), crash recovery, auto-compaction every ~5 min
- Flush thread — batched writes every 5s (transactions + balance snapshots), synced to the dashboard API
- Local mode — all execution is free when no mesh network is detected
# Check balance (requires Bearer token — own user or admin)
# Add credits (requires master API key)
Model Inference (zc infer)
# Chat model: one-shot prompt
# Speech model: transcribe an audio file (served through the hub)
-m and --file are mutually exclusive. --file accepts wav, mp3, flac, m4a,
ogg and webm; --language is an optional 2-3 letter code (^[a-z]{2,3}$). The
transcript is printed first, then a receipt (language, duration, billed
seconds, cost in zkcr). Audio longer than 10 minutes is transcribed up to that
limit and a warning says so. Hub errors (insufficient credits, model not
committed, a chat model given --file) are printed verbatim with (hub).
It uses the API URL and key saved by zc login (or ZAKURO_API_URL /
ZAKURO_API_KEY).
Benchmarking
|
| Option | Short | Default | Description |
|---|---|---|---|
--concurrency |
-c |
10 |
Concurrent workers |
--requests |
-n |
1000 |
Total requests |
--url |
-u |
http://127.0.0.1:9000 |
Broker URL |
--strategy |
-s |
best_price |
Routing strategy |
--compare |
Compare all strategies | ||
--list-strategies |
List strategies |
Output includes throughput (RPS), latency percentiles (p50-p99), success/failure rates, and an ASCII histogram.
zc bench mesh emits a JSON mesh-warmup report with per-worker latency_p95,
bandwidth_bps, and recommended_backpressure, suitable for jq or for
feeding into operator tooling.
Remote Monitoring
Read a running broker from another machine:
# transactions, request rate and latency percentiles, as JSON
System Diagnostics
Shows network info (hostname, mode, IPs), compute cluster availability (Ray, Dask, Spark), environment variables, and broker port status.
API Endpoints
The broker listens on 0.0.0.0:9000 by default.
| Method | Endpoint | Auth | Description |
|---|---|---|---|
| GET | /health |
None | Health check |
| GET | /stats |
Bearer | Broker statistics (non-admin: own txns only) |
| GET | /workers |
None | List registered workers |
| POST | /workers |
X-Worker-Key |
Register a worker |
| POST | /workers/heartbeat |
X-Worker-Key |
Worker heartbeat |
| DELETE | /workers/:id |
X-Worker-Key |
Unregister worker |
| POST | /execute |
Bearer | Execute compute request |
| POST | /price |
None | Estimate request price |
| POST | /route |
None | Placement dry run: same filtering + strategy ranking as /execute, executes/reserves/bills/mutates nothing |
| GET | /credits/:user |
Bearer | Get balance (own user or admin) |
| POST | /credits/:user/add |
X-Api-Key (master) |
Add credits |
| GET | /me |
Bearer | Current user info |
| GET | /ledger/status |
Bearer (admin) | Ledger status (API URL, connection state) |
| P2P Endpoints (ZAKURO_P2P=true) | |||
| POST | /peer/reserve |
X-Peer-Key |
Reserve credits for remote user |
| POST | /peer/commit |
X-Peer-Key |
Commit transaction after execution |
| POST | /peer/cancel |
X-Peer-Key |
Cancel reservation (refund) |
| POST | /peer/earn |
X-Peer-Key |
Credit earnings to broker owner |
| GET | /peer/balance |
X-Peer-Key |
Get balance for remote user |
| GET | /peer/workers |
X-Peer-Key |
List this broker's local workers |
| GET | /peer/health |
X-Peer-Key |
Peer broker health check |
Execute Request Options
The /execute endpoint accepts a JSON body (or X-Zakuro-Requirements header) with:
| Field | Type | Default | Description |
|---|---|---|---|
cpus |
float | 1.0 |
Required vCPUs |
memory_bytes |
int | — | Required memory in bytes |
gpus |
int | 0 |
Required GPUs |
estimated_duration_secs |
float | 1.0 |
Estimated job duration (used for cost estimation) |
budget_credits |
float | — | Maximum credits to spend; request fails if cost would exceed this |
strategy |
string | best_price |
Routing strategy |
remote_only |
bool | false |
When true, skip local workers and dispatch only to peer brokers (requires ZAKURO_P2P=true) |
worker_type |
string | — | Filter workers by type (e.g. standard, premium) |
min_vram_gb |
int | — | Minimum advertised GPU VRAM (GiB) a candidate must have; a worker that advertises none is excluded once this is set |
tags |
string[] | — | Capability tags every candidate must carry (e.g. ["apple-silicon"]). No healthy worker with all of them -> 503 NO_WORKERS_WITH_TAGS naming the missing tag; never a fallback to an untagged node |
Capability tags. A broker stamps the workers it owns with apple-silicon
when it runs on macOS/aarch64 (detected at runtime), plus any operator tags in
ZAKURO_NODE_TAGS=a,b; a worker's own /info tags are kept. Tags travel on
/workers and /peer/workers (tags, omitted when empty, ignored by older
peers), so a tagged worker on another node is placeable. Example (MLX export
step, sakura-internal#66):
-H 'X-Zakuro-Requirements: {"tags":["apple-silicon"],"cpus":2}'.
Response headers include X-Zakuro-Worker (worker ID), X-Zakuro-Cost (credits spent), and X-Zakuro-Credits-Remaining (balance after execution).
Placement Dry Run (POST /route)
Preview what /execute would do right now, for the same ResourceRequirements
JSON body /execute accepts (cpus, memory_bytes, gpus, min_vram_gb,
strategy, worker_type, tags, ...) -- without executing, reserving,
billing, or mutating any broker state (no quota reservation, no round-robin
cursor advance, no in-flight counters touched). Runs the exact router code
path /execute uses: same eligibility filtering, same strategy ranking.
|
Response (200):
A worker advertises an atelier runner endpoint via the tag
atelier-runner=<http url>; when present it is surfaced as atelier_runner
on chosen and on each candidate. When nothing is eligible, the response is
404 with {"error": ..., "code": "NO_CAPACITY", "excluded": [...]} listing
why every healthy worker was ruled out.
chosen is the policy's first eligible choice, not a dispatch guarantee:
/execute retries past a first choice that turns out to be at quota and
tries the rest before giving up. /route has no request to reserve against,
so it cannot observe or reproduce that -- chosen can name a worker
/execute ends up skipping at the moment it actually runs.
Worker Registration
Cluster Commands
| Command | Description |
|---|---|
zc up |
Start 1 worker + broker locally (foreground) |
zc up --workers N |
Start N workers + broker |
zc down |
Stop all workers + broker started by zc up |
zc workers |
List workers registered with the local broker |
zc update |
Update CLI from get.zakuro-ai.com |
zc pull |
Pull updated Docker images |
zc images |
List Zakuro images |
zc ps |
List running containers |
zc kill |
Stop running containers |
zc restart |
Restart containers |
Environment Variables
| Variable | Required | Default | Description |
|---|---|---|---|
ZAKURO_API_KEY |
No* | — | Authentication token for dashboard API. *Optional when using only local or discovered brokers (P2P only). |
ZAKURO_MASTER_KEY |
No | — | Master API key for admin operations and standalone billing. When set (without ZAKURO_API_URL), the broker runs a fully local credit ledger with billing enforced. |
ZAKURO_API_URL |
No* | — | Dashboard API URL (e.g., https://hub.zakuro-ai.com). *Required for dashboard-backed billing and worker sync. Omit to use standalone mode (ZAKURO_MASTER_KEY). |
ZAKURO_WORKER_KEY |
No | — | Shared secret for worker management endpoints (POST/DELETE /workers, heartbeat) |
ZAKURO_WIREGUARD_IP |
No | — | Override mesh IP detection (Docker sidecar setups). Named for the mesh we used to run; it applies to WireGuard too. |
ZAKURO_PEERS |
No | — | Comma-separated ip:port peers for explicit discovery (workers) or http://host:port (brokers in P2P mode) |
ZAKURO_DISCOVER_BROKER_PEERS |
No | true |
When P2P is enabled and ZAKURO_PEERS is empty, scan localhost ports 9000–9010 for other brokers. Set to false to disable. When using discovered brokers, ZAKURO_API_KEY is optional. |
ZAKURO_P2P |
No | false |
Enable P2P broker-to-broker communication |
ZAKURO_PEER_KEY |
No | — | Shared secret for P2P broker authentication (X-Peer-Key header) |
ZAKURO_OWNER_ID |
— | — | Not configurable. Always derived from ZAKURO_API_KEY (zk_{user_id}_{hex} format). Manual overrides are ignored. |
ZAKURO_NODE_NAME |
No | Auto-detected | Human-readable node name (falls back to the mesh hostname, then system hostname) |
ZAKURO_SHARE_RUNTIME |
No | auto |
Where zc share / zc up run the worker: auto (the zakuro-image-compute container when Docker is reachable, else the host uv run process with a one-line warning), host (always the host process) or container (require Docker; fails clearly without it). The container publishes its port on 127.0.0.1 only. ZC_SHARE_WORKER_IMAGE overrides the image (use name@sha256:... to pin). |
ZAKURO_WORKER_PORT |
No | 3960 |
Primary port workers listen on (discovery) |
ZAKURO_SCAN_RANGE |
No | — | Port range for discovery scan, e.g. 3960-3999 |
ZAKURO_SCAN_INTERVAL |
No | 15 |
Seconds between discovery scans |
ZAKURO_AUTO_UPDATE |
No | off | Opt-in broker self-update. 1/true/yes/on enables it; anything else is off. Never set it on k8s fleets (their image is managed by CD). When on, the broker checks the release mirror, and when a strictly newer version exists and the broker is idle (no in-flight requests, local workers or pending settlements, on 3 polls 5 s apart) it installs the sha256-verified binary and exits with code 75 so systemd (Restart=always) starts the new one. Status is in GET /health under auto_update. |
ZAKURO_AUTO_UPDATE_INTERVAL |
No | 3600 |
Seconds between update checks (+ up to 10% jitter; first check after a random 60 s..interval delay). Minimum 300. |
ZAKURO_UPDATE_URL |
No | https://get.zakuro-ai.com/zc |
Mirror base for zc update and broker auto-update. Either a bare host base (https://stg.get.zakuro-ai.com) or one ending in /zc; trailing / is trimmed and /zc appended when missing. Must be https:// (plain http:// only for 127.0.0.1/localhost); an invalid value is an error, never silently ignored. |
Development
Task Commands
This project uses Task as its task runner. Install it with go install github.com/go-task/task/v3/cmd/task@latest or see installation docs.
| Command | Description |
|---|---|
task build |
Build binary and start Docker services |
task build:bin |
Incremental release build of zc and zc-hooks, installed to ~/.local/bin |
task dev |
Incremental debug build for fixing bugs, linked as ~/.local/bin/zc-dev |
task build:docker |
Build Docker image only |
task run |
Build and run the CLI |
task debug |
Build and run with RUST_BACKTRACE=full |
task release |
Run the installed release binary |
task broker |
Start broker in foreground |
task broker:daemon |
Start broker in daemon mode |
task bench |
Run default benchmark |
task bench:compare |
Compare all routing strategies |
task info |
Show system diagnostics |
task docker:up |
Start Docker services |
task docker:down |
Stop Docker services |
task docker:logs |
Tail Docker service logs |
task clean |
Remove build artifacts |
Docker
# Build and run with Docker Compose
# Or build the image directly
Project Structure
src/
main.rs # CLI entry point and command routing
broker/
mod.rs # Broker state, config, and per-worker local detection
server.rs # HTTP server (tiny_http) with execute handler
worker.rs # Worker registry, resources, and health tracking
router.rs # Routing strategies and worker selection
credits.rs # Lightweight per-user balance cache and rate-limit tracker (reservation/history owned by Ledger)
ledger.rs # Central ledger: API mode (dashboard) or local in-memory reserve-commit
peer.rs # P2P broker-to-broker communication (PeerClient, PeerManager, FNV-1a authority)
flush.rs # TransactionBuffer for batched writes (5s cycle, via dashboard API)
discovery.rs # mesh/local/peer-based worker discovery
wal.rs # Write-ahead log (JSONL + DashMap index, batched fsync)
recovery.rs # WAL replay on startup (reserved→cancel, executed→commit)
bench.rs # Benchmarking tool
stats.rs # Statistics tracking
info.rs # System diagnostics
common/mod.rs # Shared utilities
exec/mod.rs # Command execution helpers
manager/mod.rs # Docker container management
network/mod.rs # Network interface detection
envs/mod.rs # Environment variable handling
Troubleshooting
- "You are missing ZAKURO_API_KEY" — set
ZAKURO_API_KEYfor dashboard auth. For standalone billing, setZAKURO_MASTER_KEYinstead. P2P-only setups (auto-discovered peers) don't require either. - No workers discovered — ensure workers expose
/healthand/infoendpoints on port 3960 (orZAKURO_WORKER_PORT). Check withzc info. - Credit operations failing — ensure both
ZAKURO_API_KEYandZAKURO_API_URLare set for dashboard mode. For standalone mode, set onlyZAKURO_MASTER_KEY. Verify the dashboard returns thecredits_balancefield in/api/broker/balance/{id}. - Zero cost on remote executions — ensure
is_billing_enabled()returns true: either dashboard URL+key orZAKURO_MASTER_KEYmust be set. Local workers (127.0.0.1/localhost) are always free; remote mesh workers are billed. remote_only: truereturns 503 — ensureZAKURO_P2P=trueand at least one peer broker is reachable. Without peers,remote_onlyrequests cannot be served.- P2P mode not working — ensure
ZAKURO_P2P=true,ZAKURO_PEER_KEYmatches on all brokers. UseZAKURO_PEERSfor explicit URLs, or leave empty to auto-discover brokers on localhost (ports 9000–9010). - Mesh IP not detected — the broker recognises a
zakuro0orwg*interface holding an address in10.13.13.0/24(seeget_mesh_ipinsrc/broker/discovery.rs). An interface outside that subnet is ignored. SetZAKURO_WIREGUARD_IPto override.
Documentation
See docs/USAGE.md for the complete usage guide covering all API details, configuration reference, and the credit system in depth.
See docs/QUIC_TRANSPORT.md for details on the QUIC-based P2P transport layer.
License
MIT