<div align="center">
<h1>zc</h1>
<p>P2P compute broker with credit-based billing, automatic worker discovery, and real-time monitoring.</p>
</div>
<p align="center">
<a href="#overview">Overview</a> •
<a href="#installation">Installation</a> •
<a href="#quick-start">Quick Start</a> •
<a href="#zakuro-drive-hooks">Drive Hooks</a> •
<a href="#broker">Broker</a> •
<a href="#benchmarking">Benchmarking</a> •
<a href="#api-endpoints">API Endpoints</a> •
<a href="#development">Development</a>
</p>
## Overview
`zc` is the Zakuro AI command-line tool that acts as a P2P compute broker. It routes compute requests to workers across a distributed network, handles credit-based billing via the Zakuro dashboard API with write-ahead logging, automatically discovers workers on the WireGuard mesh, and provides real-time monitoring through an interactive TUI dashboard.
**Key features:**
- **P2P compute routing** — forward requests to optimal workers based on price, latency, or availability
- **P2P broker mesh** — broker-to-broker communication with authoritative user assignment (ZAKURO_P2P mode)
- **QUIC transport** — high-throughput multiplexed UDP transport for peer task offers (QUIC-first with HTTP fallback)
- **6 routing strategies** — `best_price`, `best_latency`, `best_availability`, `round_robin`, `random`, `weighted_capacity`
- **Credit-based billing** — dashboard API-backed ledger with atomic reserve-commit transactions; no direct database access; standalone mode (`ZAKURO_MASTER_KEY`) runs a local ledger without dashboard
- **Write-ahead log** — JSONL WAL with DashMap in-memory index, batched fsync, and crash recovery
- **Zero-latency hot path** — in-memory DashMap credit operations for authoritative users (P2P mode)
- **Automatic discovery** — finds workers on the mesh subnet (parallel scan) or falls back to localhost
- **Interactive TUI** — real-time dashboard showing transactions, workers, metrics, and RPS sparklines
- **Built-in benchmarking** — measure throughput/latency and compare routing strategies
- **Cluster management** — Docker container lifecycle, image management, and diagnostics
- **Zakuro Drive hooks** — manage file-event hooks (webhook/exec) on a local [zakuro-drive](https://github.com/zakuro-ai/zakuro-drive) daemon over its QUIC API
## Installation
### Prerequisites
- [Rust](https://rustup.rs/) — recent stable (the dependency tree's MSRV is 1.88 via `time` 0.3.47; the old "1.75+" note predates that)
- [Task](https://taskfile.dev/) (replaces Make)
- Docker and Docker Compose (optional, for containerized deployments)
- A Zakuro API key (`ZAKURO_API_KEY`) — sign up at [zakuro-ai.com](https://zakuro-ai.com)
### From Source
```bash
git clone https://github.com/zakuro-ai/zc.git
cd zc
task build:bin
```
This compiles the release binary and installs it to `/usr/local/bin/zc`.
#### Install with Cargo (no clone)
```bash
cargo install --git https://github.com/zakuro-ai/zc zc2
```
`zc2` needs **no access to any private repository** — it simply does not
include `zc hooks` (Zakuro Drive file-event hooks). That command lives in a
separate `zc-hooks` helper binary which `zc hooks` execs, because it depends on
the private `zakuro-client` crate. Building it requires read access to
`zakuro-ai/zakuro-drive`:
```bash
cargo build --release -p zc-hooks
```
Put the resulting `zc-hooks` next to your `zc` binary (or anywhere on `PATH`)
and `zc hooks …` works as before.
### From crates.io
```bash
cargo install zc2 --locked
```
> **`--locked` is required, not optional.** `tui-textarea 0.4` declares
> `ratatui >=0.23.0, <1`, so a fresh dependency resolution satisfies it with
> ratatui 0.30 while `zc2` itself pins ratatui 0.26 — two incompatible copies,
> and the build fails with
> `expected ratatui_core::style::Style, found ratatui::style::Style`.
> `--locked` uses the lockfile shipped in the published crate, which pins a
> single ratatui 0.26.3. Tracked in #196.
Note the registry build has no `zc hooks` — that command needs the `zc-hooks`
sidecar, which depends on a private crate and so cannot be published. Use a
[binary release](https://github.com/zakuro-ai/zc/releases) if you need it.
### With Docker
```bash
task build
```
Builds the binary, then builds and starts the Docker services via `docker compose`.
### Verify
```bash
zc --version
```
## Quick Start
```bash
# 1. Set your API key
export ZAKURO_API_KEY="zk_myuser_abc123"
# 2. Set dashboard API URL
export ZAKURO_API_URL="https://my.zakuro-ai.com"
# 3. Start the broker with interactive TUI
zc -t broker
# The broker discovers workers, listens on 0.0.0.0:9000,
# and routes all credit operations through the dashboard API.
```
## Zakuro Drive Hooks
```bash
# Webhook hook — secret read from stdin (recommended; avoids ps/history leaks)
echo "$WEBHOOK_SECRET" | zc hooks add --name slack-notify \
--on file.created --on file.modified --glob "reports/**/*.pdf" \
--webhook https://hooks.example.com/zakuro --secret-stdin
# Local command hook with a custom timeout
zc hooks add --name csv-to-parquet --on file.created --glob "**/*.csv" \
--exec "/usr/local/bin/convert-csv" --timeout 120
zc hooks list --all # include disabled hooks
zc hooks get csv-to-parquet # full config (secrets redacted by the daemon)
zc hooks update csv-to-parquet --disable
zc hooks logs --failed --from 2h
zc hooks remove csv-to-parquet
```
The daemon address defaults to `127.0.0.1:4719`; override with `--daemon HOST:PORT` or `ZAKURO_DRIVE_DAEMON`. Run `zc hooks --help` for the full flag reference (retry tuning, `--secret-from-env`, `--tls-insecure`, dead-letter views).
> **Packaging note:** `zakuro-client` (the drive daemon API client) is consumed as a git dependency on the private `zakuro-ai/zakuro-drive` repository, pinned to the `client-v0.1.0` tag. crates.io rejects git dependencies, so the crates.io publish job in `auto-release.yml` is paused until `zakuro-proto` + `zakuro-client` are published to a registry; GitHub releases with prebuilt binaries continue.
## Broker
The broker accepts compute requests, selects an optimal worker, forwards the request, and handles billing.
### Modes
| Foreground | `zc broker` | Live transaction log with colored output |
| TUI | `zc -t broker` | Interactive dashboard (workers, metrics, transactions) |
| Daemon | `zc -d broker` | Background process, minimal output |
```bash
zc broker # foreground on :9000
zc -t broker # TUI on :9000
zc -d broker # daemon on :9000
zc broker 8080 # custom port
zc broker 0.0.0.0 9000 # custom host and port
```
### TUI Dashboard
The TUI (`-t` / `--tui`) provides four panels:
- **Header** — mode (P2P/LOCAL), ledger status, uptime, listen address
- **Transactions** — scrollable live log with status, user, cost, latency
- **Workers** — connected workers with resources, pricing, health
- **Metrics** — total requests, RPS sparkline, latency percentiles (p50/p90/p99), credits spent
| `Tab` | Switch panels |
| `j`/`k` `Up`/`Down` | Scroll |
| `g` / `G` | Top / Bottom |
| `?` | Help overlay |
| `q` / `Esc` | Quit |
### Worker Discovery
| **Mesh** | a `zakuro0` or `wg*` interface with a `10.13.13.x` address, or `ZAKURO_WIREGUARD_IP` env | Scans peers + subnet; credits enforced for remote workers |
| **Peers** | `ZAKURO_PEERS` env set (no mesh interface) | Probes explicit peer addresses; credits enforced |
| **Local** | No mesh, no peers | Scans `localhost:8089-8091`; execution is free |
Workers must expose `/health` and `/info` endpoints to be discovered. Workers can also [register manually](#post-workers) via the API.
In a **P2P mesh**, each broker detects its own mesh IP and treats workers at `localhost`/`127.0.0.1` or at that IP as **local** (free execution). Remote workers (mesh peers) are charged via the credit ledger. See [docs/USAGE.md](docs/USAGE.md#p2p-mesh-deployment) for mesh setup instructions.
Subnet scanning is **parallelized**: phase 1 does a concurrent TCP SYN to all 253 /24 hosts (50ms timeout); phase 2 probes only reachable hosts. This reduces scan time from `253 × 50ms` to `~50ms + probe time`.
### P2P Broker-to-Broker Communication
Enable distributed broker mesh with `ZAKURO_P2P=true`. Each broker becomes authoritative for a subset of users:
| **Authority::Local** | User owned by this broker (via FNV-1a hash) → in-memory DashMap, zero PG on hot path |
| **Authority::Peer(url)** | User owned by remote broker → HTTP `/peer/reserve` + `/peer/commit` (~1ms RTT) |
| **Authority::Standalone** | Peer unreachable → dashboard API fallback path |
**Peer endpoints** (`X-Peer-Key` auth):
- `POST /peer/reserve` — reserve credits for remote user
- `POST /peer/commit` — commit transaction after execution
- `POST /peer/cancel` — cancel reservation (refund)
- `GET /peer/balance` — get balance for remote user
- `GET /peer/health` — peer health check
**Environment variables:**
- `ZAKURO_P2P=true` — enable P2P mode (default: false)
- `ZAKURO_PEER_KEY` — shared secret for peer authentication
- `ZAKURO_OWNER_ID` — this broker's ID for authority assignment
- `ZAKURO_PEERS` — comma-separated peer broker URLs for discovery
### Routing Strategies
| `best_price` | Cheapest worker **(default)** |
| `best_latency` | Fastest response time |
| `best_availability` | Most available resources |
| `round_robin` | Even distribution |
| `random` | Random selection |
| `weighted_capacity` | Weighted by available CPU/memory |
Set via the `strategy` field in `X-Zakuro-Requirements` header on `/execute` requests.
### Credit System
- **Dashboard API ledger** — all balance and transaction operations go through `ZAKURO_API_URL`; no direct database access from the broker. The dashboard returns balance in the `credits_balance` field.
- **Standalone mode** — set `ZAKURO_MASTER_KEY` (no `ZAKURO_API_URL` needed) to run a fully local credit ledger. Transactions are queued in the WAL and the flush buffer; billing is enforced even without the dashboard.
- **P2P mode** (`ZAKURO_P2P=true`) — authoritative broker owns users via FNV-1a hash, DashMap in-memory operations on the hot path. Setting `remote_only: true` in a request forces routing to peer workers (bypasses local worker selection).
- **Write-ahead log** — append-only JSONL with DashMap index (O(1) lookup), batched fsync (100ms/64 entries), crash recovery, auto-compaction every ~5 min
- **Flush thread** — batched writes every 5s (transactions + balance snapshots), synced to the dashboard API
- **Local mode** — all execution is free when no mesh network is detected
```bash
# Check balance (requires Bearer token — own user or admin)
curl http://localhost:9000/credits/my-user \
-H "Authorization: Bearer $API_KEY"
# Add credits (requires master API key)
curl -X POST http://localhost:9000/credits/my-user/add \
-H "Content-Type: application/json" \
-H "X-Api-Key: $ZAKURO_MASTER_KEY" \
-d '{"amount": 100.0, "description": "Initial deposit"}'
```
## Benchmarking
```bash
zc bench # default (10 concurrent, 1000 requests)
zc bench -c 100 -n 10000 # high-load
zc bench -s best_latency # specific strategy
zc bench --compare # compare all strategies
zc bench http://broker:9000 # remote broker
| `--concurrency` | `-c` | `10` | Concurrent workers |
| `--requests` | `-n` | `1000` | Total requests |
| `--url` | `-u` | `http://127.0.0.1:9000` | Broker URL |
| `--strategy` | `-s` | `best_price` | Routing strategy |
| `--compare` | | | Compare all strategies |
| `--list-strategies` | | | List strategies |
Output includes throughput (RPS), latency percentiles (p50-p99), success/failure rates, and an ASCII histogram.
`zc bench mesh` emits a JSON mesh-warmup report with per-worker `latency_p95`,
`bandwidth_bps`, and `recommended_backpressure`, suitable for `jq` or for
feeding into operator tooling.
## Remote Monitoring
Attach to a running broker from another machine:
```bash
zc attach localhost:9000
zc attach http://broker.mesh:9000
```
Renders the same TUI dashboard using the broker's `/stats` endpoint. Press `r` to force refresh.
## System Diagnostics
```bash
zc info
```
Shows network info (hostname, mode, IPs), compute cluster availability (Ray, Dask, Spark), environment variables, and broker port status.
## API Endpoints
The broker listens on `0.0.0.0:9000` by default.
| GET | `/health` | None | Health check |
| GET | `/stats` | Bearer | Broker statistics (non-admin: own txns only) |
| GET | `/workers` | None | List registered workers |
| POST | `/workers` | `X-Worker-Key` | Register a worker |
| POST | `/workers/heartbeat` | `X-Worker-Key` | Worker heartbeat |
| DELETE | `/workers/:id` | `X-Worker-Key` | Unregister worker |
| POST | `/execute` | Bearer | Execute compute request |
| POST | `/price` | None | Estimate request price |
| GET | `/credits/:user` | Bearer | Get balance (own user or admin) |
| POST | `/credits/:user/add` | `X-Api-Key` (master) | Add credits |
| GET | `/me` | Bearer | Current user info |
| GET | `/ledger/status` | Bearer (admin) | Ledger status (API URL, connection state) |
| **P2P Endpoints** (ZAKURO_P2P=true) | | | |
| POST | `/peer/reserve` | `X-Peer-Key` | Reserve credits for remote user |
| POST | `/peer/commit` | `X-Peer-Key` | Commit transaction after execution |
| POST | `/peer/cancel` | `X-Peer-Key` | Cancel reservation (refund) |
| POST | `/peer/earn` | `X-Peer-Key` | Credit earnings to broker owner |
| GET | `/peer/balance` | `X-Peer-Key` | Get balance for remote user |
| GET | `/peer/workers` | `X-Peer-Key` | List this broker's local workers |
| GET | `/peer/health` | `X-Peer-Key` | Peer broker health check |
### Execute Request Options
The `/execute` endpoint accepts a JSON body (or `X-Zakuro-Requirements` header) with:
| `cpus` | float | `1.0` | Required vCPUs |
| `memory_bytes` | int | — | Required memory in bytes |
| `gpus` | int | `0` | Required GPUs |
| `estimated_duration_secs` | float | `1.0` | Estimated job duration (used for cost estimation) |
| `budget_credits` | float | — | Maximum credits to spend; request fails if cost would exceed this |
| `strategy` | string | `best_price` | Routing strategy |
| `remote_only` | bool | `false` | When `true`, skip local workers and dispatch only to peer brokers (requires `ZAKURO_P2P=true`) |
| `worker_type` | string | — | Filter workers by type (e.g. `standard`, `premium`) |
Response headers include `X-Zakuro-Worker` (worker ID), `X-Zakuro-Cost` (credits spent), and `X-Zakuro-Credits-Remaining` (balance after execution).
### Worker Registration
```json
{
"name": "worker-1",
"uri": "http://10.13.13.5:8089",
"worker_type": "zakuro",
"resources": {
"cpus_total": 8.0,
"cpus_available": 8.0,
"memory_total": 17179869184,
"memory_available": 17179869184,
"gpus_total": 1,
"gpus_available": 1
},
"pricing": {
"cpu_price": 0.001,
"memory_price": 0.0001,
"gpu_price": 0.01,
"min_charge": 0.001
}
}
```
## Cluster Commands
| `zc up` | Start 1 worker + broker locally (foreground) |
| `zc up --workers N` | Start N workers + broker |
| `zc down` | Stop all workers + broker started by `zc up` |
| `zc workers` | List workers registered with the local broker |
| `zc update` | Update CLI from get.zakuro-ai.com |
| `zc pull` | Pull updated Docker images |
| `zc images` | List Zakuro images |
| `zc ps` | List running containers |
| `zc kill` | Stop running containers |
| `zc restart` | Restart containers |
## Environment Variables
| `ZAKURO_API_KEY` | No* | — | Authentication token for dashboard API. *Optional when using only local or discovered brokers (P2P only). |
| `ZAKURO_MASTER_KEY` | No | — | Master API key for admin operations and standalone billing. When set (without `ZAKURO_API_URL`), the broker runs a fully local credit ledger with billing enforced. |
| `ZAKURO_API_URL` | No* | — | Dashboard API URL (e.g., `https://my.zakuro-ai.com`). *Required for dashboard-backed billing and worker sync. Omit to use standalone mode (`ZAKURO_MASTER_KEY`). |
| `ZAKURO_WORKER_KEY` | No | — | Shared secret for worker management endpoints (`POST/DELETE /workers`, heartbeat) |
| `ZAKURO_WIREGUARD_IP` | No | — | Override mesh IP detection (Docker sidecar setups). Named for the mesh we used to run; it applies to WireGuard too. |
| `ZAKURO_PEERS` | No | — | Comma-separated `ip:port` peers for explicit discovery (workers) or `http://host:port` (brokers in P2P mode) |
| `ZAKURO_DISCOVER_BROKER_PEERS` | No | `true` | When P2P is enabled and `ZAKURO_PEERS` is empty, scan localhost ports 9000–9010 for other brokers. Set to `false` to disable. When using discovered brokers, `ZAKURO_API_KEY` is optional. |
| `ZAKURO_P2P` | No | `false` | Enable P2P broker-to-broker communication |
| `ZAKURO_PEER_KEY` | No | — | Shared secret for P2P broker authentication (`X-Peer-Key` header) |
| `ZAKURO_OWNER_ID` | — | — | **Not configurable.** Always derived from `ZAKURO_API_KEY` (`zk_{user_id}_{hex}` format). Manual overrides are ignored. |
| `ZAKURO_NODE_NAME` | No | Auto-detected | Human-readable node name (falls back to the mesh hostname, then system hostname) |
| `ZAKURO_WORKER_PORT` | No | `3960` | Primary port workers listen on (discovery) |
| `ZAKURO_SCAN_RANGE` | No | — | Port range for discovery scan, e.g. `3960-3999` |
| `ZAKURO_SCAN_INTERVAL` | No | `15` | Seconds between discovery scans |
## Development
### Task Commands
This project uses [Task](https://taskfile.dev/) as its task runner. Install it with `go install github.com/go-task/task/v3/cmd/task@latest` or see [installation docs](https://taskfile.dev/installation/).
| `task build` | Build binary and start Docker services |
| `task build:bin` | Compile release binary and install to `/usr/local/bin/zc` |
| `task build:docker` | Build Docker image only |
| `task run` | Build and run the CLI |
| `task debug` | Build and run with `RUST_BACKTRACE=full` |
| `task release` | Run the installed release binary |
| `task broker` | Start broker in foreground |
| `task broker:tui` | Start broker with TUI dashboard |
| `task broker:daemon` | Start broker in daemon mode |
| `task bench` | Run default benchmark |
| `task bench:compare` | Compare all routing strategies |
| `task info` | Show system diagnostics |
| `task docker:up` | Start Docker services |
| `task docker:down` | Stop Docker services |
| `task docker:logs` | Tail Docker service logs |
| `task clean` | Remove build artifacts |
### Docker
```bash
# Build and run with Docker Compose
task build
# Or build the image directly
docker build -t zakuro-ai/broker -f docker/Dockerfile .
docker run -p 9000:9000 -e ZAKURO_API_KEY=demo zakuro-ai/broker
```
### Project Structure
```
src/
main.rs # CLI entry point and command routing
broker/
mod.rs # Broker state, config, and per-worker local detection
server.rs # HTTP server (tiny_http) with execute handler
worker.rs # Worker registry, resources, and health tracking
router.rs # Routing strategies and worker selection
credits.rs # Lightweight per-user balance cache and rate-limit tracker (reservation/history owned by Ledger)
ledger.rs # Central ledger: API mode (dashboard) or local in-memory reserve-commit
peer.rs # P2P broker-to-broker communication (PeerClient, PeerManager, FNV-1a authority)
flush.rs # TransactionBuffer for batched writes (5s cycle, via dashboard API)
discovery.rs # mesh/local/peer-based worker discovery
wal.rs # Write-ahead log (JSONL + DashMap index, batched fsync)
recovery.rs # WAL replay on startup (reserved→cancel, executed→commit)
tui.rs # Interactive terminal dashboard (ratatui)
bench.rs # Benchmarking tool
stats.rs # Statistics tracking
info.rs # System diagnostics
common/mod.rs # Shared utilities
exec/mod.rs # Command execution helpers
manager/mod.rs # Docker container management
network/mod.rs # Network interface detection
envs/mod.rs # Environment variable handling
```
## Troubleshooting
- **"You are missing ZAKURO_API_KEY"** — set `ZAKURO_API_KEY` for dashboard auth. For standalone billing, set `ZAKURO_MASTER_KEY` instead. P2P-only setups (auto-discovered peers) don't require either.
- **No workers discovered** — ensure workers expose `/health` and `/info` endpoints on port 3960 (or `ZAKURO_WORKER_PORT`). Check with `zc info`.
- **Credit operations failing** — ensure both `ZAKURO_API_KEY` and `ZAKURO_API_URL` are set for dashboard mode. For standalone mode, set only `ZAKURO_MASTER_KEY`. Verify the dashboard returns the `credits_balance` field in `/api/broker/balance/{id}`.
- **Zero cost on remote executions** — ensure `is_billing_enabled()` returns true: either dashboard URL+key or `ZAKURO_MASTER_KEY` must be set. Local workers (`127.0.0.1`/`localhost`) are always free; remote mesh workers are billed.
- **`remote_only: true` returns 503** — ensure `ZAKURO_P2P=true` and at least one peer broker is reachable. Without peers, `remote_only` requests cannot be served.
- **P2P mode not working** — ensure `ZAKURO_P2P=true`, `ZAKURO_PEER_KEY` matches on all brokers. Use `ZAKURO_PEERS` for explicit URLs, or leave empty to auto-discover brokers on localhost (ports 9000–9010).
- **Mesh IP not detected** — the broker recognises a `zakuro0` or `wg*` interface holding an address in `10.13.13.0/24` (see `get_mesh_ip` in `src/broker/discovery.rs`). An interface outside that subnet is ignored. Set `ZAKURO_WIREGUARD_IP` to override.
- **TUI not rendering** — ensure your terminal supports 256 colors and is at least 80 columns wide.
## Documentation
See [docs/USAGE.md](docs/USAGE.md) for the complete usage guide covering all API details, configuration reference, and the credit system in depth.
See [docs/QUIC_TRANSPORT.md](docs/QUIC_TRANSPORT.md) for details on the QUIC-based P2P transport layer.
## License
MIT