<div align="center">
<h1>zc</h1>
<p>P2P compute broker with credit-based billing, automatic worker discovery, and real-time monitoring.</p>
</div>
<p align="center">
<a href="#overview">Overview</a> •
<a href="#installation">Installation</a> •
<a href="#quick-start">Quick Start</a> •
<a href="#broker">Broker</a> •
<a href="#benchmarking">Benchmarking</a> •
<a href="#api-endpoints">API Endpoints</a> •
<a href="#development">Development</a>
</p>
## Overview
`zc` is the Zakuro AI command-line tool that acts as a P2P compute broker. It routes compute requests to workers across a distributed network, handles credit-based billing via a PostgreSQL ledger with write-ahead logging, automatically discovers workers on Tailscale networks, and provides real-time monitoring through an interactive TUI dashboard.
**Key features:**
- **P2P compute routing** — forward requests to optimal workers based on price, latency, or availability
- **P2P broker mesh** — broker-to-broker communication with authoritative user assignment (ZAKURO_P2P mode)
- **QUIC transport** — high-throughput multiplexed UDP transport for peer task offers (QUIC-first with HTTP fallback)
- **6 routing strategies** — `best_price`, `best_latency`, `best_availability`, `round_robin`, `random`, `weighted_capacity`
- **Credit-based billing** — PostgreSQL-backed ledger with atomic reserve-commit transactions, or API mode (dashboard API, no direct PG access)
- **Write-ahead log** — JSONL WAL with DashMap in-memory index, batched fsync, and crash recovery
- **Zero-PG hot path** — in-memory DashMap credit operations for authoritative users (P2P mode)
- **Automatic discovery** — finds workers on Tailscale/WireGuard subnets or falls back to localhost
- **Interactive TUI** — real-time dashboard showing transactions, workers, metrics, and RPS sparklines
- **Built-in benchmarking** — measure throughput/latency and compare routing strategies
- **Cluster management** — Docker container lifecycle, image management, and diagnostics
## Installation
### Prerequisites
- [Rust](https://rustup.rs/) 1.75+
- [Task](https://taskfile.dev/) (replaces Make)
- Docker and Docker Compose (optional, for containerized deployments)
- PostgreSQL (optional, for centralized credit ledger with restricted broker role)
### From Source
```bash
git clone https://github.com/zakuro-ai/zc.git
cd zc
task build:bin
```
This compiles the release binary and installs it to `/usr/local/bin/zc`.
### With Docker
```bash
task build
```
Builds the binary, then builds and starts the Docker services via `docker compose`.
### Verify
```bash
zc --version
```
## Quick Start
### Local Development Mode
```bash
# 1. Set your auth token
export ZAKURO_API_KEY="your-token"
# 2. Set PostgreSQL connection (for local credit ledger)
export DATABASE_URL="postgresql://zakuro_broker:broker_secret@localhost:5432/zakuro"
# 3. Start the broker with interactive TUI
zc -t broker
# 4. The broker discovers workers and listens on 0.0.0.0:9000
```
### Production / API Mode
When both `ZAKURO_API_URL` and `ZAKURO_API_KEY` are set the broker runs in
**API mode**: all credit and worker-sync operations go through the dashboard
API and the PostgreSQL connection pool is **never created**.
```bash
# 1. Set your API key
export ZAKURO_API_KEY="zk_myuser_abc123"
# 2. Set dashboard API URL
export ZAKURO_API_URL="https://my.zakuro-ai.com"
# 3. Start the broker
zc -t broker
# Credit operations use dashboard API (no PostgreSQL access)
# Tailscale auth key automatically retrieved from dashboard API
```
**Benefits of API mode:**
- ✅ Single credential (`ZAKURO_API_KEY`)
- ✅ No PostgreSQL credentials or connection in the broker
- ✅ Tailscale auth key auto-fetched
- ✅ Centralized configuration management
- ✅ Revocable API keys
## Broker
The broker accepts compute requests, selects an optimal worker, forwards the request, and handles billing.
### Modes
| Foreground | `zc broker` | Live transaction log with colored output |
| TUI | `zc -t broker` | Interactive dashboard (workers, metrics, transactions) |
| Daemon | `zc -d broker` | Background process, minimal output |
```bash
zc broker # foreground on :9000
zc -t broker # TUI on :9000
zc -d broker # daemon on :9000
zc broker 8080 # custom port
zc broker 0.0.0.0 9000 # custom host and port
```
### TUI Dashboard
The TUI (`-t` / `--tui`) provides four panels:
- **Header** — mode (P2P/LOCAL), ledger status, uptime, listen address
- **Transactions** — scrollable live log with status, user, cost, latency
- **Workers** — connected workers with resources, pricing, health
- **Metrics** — total requests, RPS sparkline, latency percentiles (p50/p90/p99), credits spent
| `Tab` | Switch panels |
| `j`/`k` `Up`/`Down` | Scroll |
| `g` / `G` | Top / Bottom |
| `?` | Help overlay |
| `q` / `Esc` | Quit |
### Worker Discovery
| **Tailscale** | `tailscale0`/`wg0`/`ts*` interface or `ZAKURO_TAILSCALE_IP` env | Scans peers + subnet; credits enforced for remote workers |
| **Peers** | `ZAKURO_PEERS` env set (no Tailscale interface) | Probes explicit peer addresses; credits enforced |
| **Local** | No Tailscale, no peers | Scans `localhost:8089-8091`; execution is free |
Workers must expose `/health` and `/info` endpoints to be discovered. Workers can also [register manually](#post-workers) via the API.
In a **P2P mesh**, each broker detects its own Tailscale IP and treats workers at `localhost` or that IP as **local** (free execution). Remote workers are charged via the credit ledger. See [docs/USAGE.md](docs/USAGE.md#p2p-mesh-deployment) for mesh setup instructions.
### P2P Broker-to-Broker Communication
Enable distributed broker mesh with `ZAKURO_P2P=true`. Each broker becomes authoritative for a subset of users:
| **Authority::Local** | User owned by this broker (via FNV-1a hash) → in-memory DashMap, zero PG on hot path |
| **Authority::Peer(url)** | User owned by remote broker → HTTP `/peer/reserve` + `/peer/commit` (~1ms RTT) |
| **Authority::Standalone** | Peer unreachable → existing PG / API fallback path |
**Peer endpoints** (`X-Peer-Key` auth):
- `POST /peer/reserve` — reserve credits for remote user
- `POST /peer/commit` — commit transaction after execution
- `POST /peer/cancel` — cancel reservation (refund)
- `GET /peer/balance` — get balance for remote user
- `GET /peer/health` — peer health check
**Environment variables:**
- `ZAKURO_P2P=true` — enable P2P mode (default: false)
- `ZAKURO_PEER_KEY` — shared secret for peer authentication
- `ZAKURO_OWNER_ID` — this broker's ID for authority assignment
- `ZAKURO_PEERS` — comma-separated peer broker URLs for discovery
### Routing Strategies
| `best_price` | Cheapest worker **(default)** |
| `best_latency` | Fastest response time |
| `best_availability` | Most available resources |
| `round_robin` | Even distribution |
| `random` | Random selection |
| `weighted_capacity` | Weighted by available CPU/memory |
Set via the `strategy` field in `X-Zakuro-Requirements` header on `/execute` requests.
### Credit System
- **PostgreSQL ledger** — stored procedures with `SECURITY DEFINER`, restricted `zakuro_broker` PG role with zero direct table access
- **API mode** — when `ZAKURO_API_URL` + `ZAKURO_API_KEY` are set, PG is never accessed directly; all operations go through the dashboard API
- **P2P mode** (ZAKURO_P2P=true) — authoritative broker owns users via FNV-1a hash, DashMap in-memory operations, zero PG on hot path
- **Write-ahead log** — append-only JSONL with DashMap index (O(1) lookup), batched fsync (100ms/64 entries), crash recovery, auto-compaction every ~5 min
- **Flush thread** — batched writes every 5s (transactions + balance snapshots) in P2P mode, via API or PG
- **Local fallback** — in-memory when PostgreSQL is unavailable
- **Local mode** — all execution is free when no Tailscale network is detected
```bash
# Check balance (requires Bearer token — own user or admin)
curl http://localhost:9000/credits/my-user \
-H "Authorization: Bearer $API_KEY"
# Add credits (requires master API key)
curl -X POST http://localhost:9000/credits/my-user/add \
-H "Content-Type: application/json" \
-H "X-Api-Key: $ZAKURO_MASTER_KEY" \
-d '{"amount": 100.0, "description": "Initial deposit"}'
# Check ledger status (admin only, password redacted)
curl http://localhost:9000/ledger/status \
-H "Authorization: Bearer $ZAKURO_MASTER_KEY"
```
## Benchmarking
```bash
zc bench # default (10 concurrent, 1000 requests)
zc bench -c 100 -n 10000 # high-load
zc bench -s best_latency # specific strategy
zc bench --compare # compare all strategies
zc bench http://broker:9000 # remote broker
```
| `--concurrency` | `-c` | `10` | Concurrent workers |
| `--requests` | `-n` | `1000` | Total requests |
| `--url` | `-u` | `http://127.0.0.1:9000` | Broker URL |
| `--strategy` | `-s` | `best_price` | Routing strategy |
| `--compare` | | | Compare all strategies |
| `--list-strategies` | | | List strategies |
Output includes throughput (RPS), latency percentiles (p50-p99), success/failure rates, and an ASCII histogram.
## Remote Monitoring
Attach to a running broker from another machine:
```bash
zc attach localhost:9000
zc attach http://broker.tailnet:9000
```
Renders the same TUI dashboard using the broker's `/stats` endpoint. Press `r` to force refresh.
## System Diagnostics
```bash
zc info
```
Shows network info (hostname, mode, IPs), compute cluster availability (Ray, Dask, Spark), environment variables, and broker port status.
## API Endpoints
The broker listens on `0.0.0.0:9000` by default.
| GET | `/health` | None | Health check |
| GET | `/stats` | Bearer | Broker statistics (non-admin: own txns only) |
| GET | `/workers` | None | List registered workers |
| POST | `/workers` | `X-Worker-Key` | Register a worker |
| POST | `/workers/heartbeat` | `X-Worker-Key` | Worker heartbeat |
| DELETE | `/workers/:id` | `X-Worker-Key` | Unregister worker |
| POST | `/execute` | Bearer | Execute compute request |
| POST | `/price` | None | Estimate request price |
| GET | `/credits/:user` | Bearer | Get balance (own user or admin) |
| POST | `/credits/:user/add` | `X-Api-Key` (master) | Add credits |
| GET | `/me` | Bearer | Current user info |
| GET | `/ledger/status` | Bearer (admin) | Ledger connection status (password redacted) |
| **P2P Endpoints** (ZAKURO_P2P=true) | | | |
| POST | `/peer/reserve` | `X-Peer-Key` | Reserve credits for remote user |
| POST | `/peer/commit` | `X-Peer-Key` | Commit transaction after execution |
| POST | `/peer/cancel` | `X-Peer-Key` | Cancel reservation (refund) |
| POST | `/peer/earn` | `X-Peer-Key` | Credit earnings to broker owner |
| GET | `/peer/balance` | `X-Peer-Key` | Get balance for remote user |
| GET | `/peer/workers` | `X-Peer-Key` | List this broker's local workers |
| GET | `/peer/health` | `X-Peer-Key` | Peer broker health check |
### Worker Registration
```json
{
"name": "worker-1",
"uri": "http://10.13.13.5:8089",
"worker_type": "zakuro",
"resources": {
"cpus_total": 8.0,
"cpus_available": 8.0,
"memory_total": 17179869184,
"memory_available": 17179869184,
"gpus_total": 1,
"gpus_available": 1
},
"pricing": {
"cpu_price": 0.001,
"memory_price": 0.0001,
"gpu_price": 0.01,
"min_charge": 0.001
}
}
```
## Cluster Commands
| `zc up` | Start 1 worker + broker locally (foreground) |
| `zc up --workers N` | Start N workers + broker |
| `zc down` | Stop all workers + broker started by `zc up` |
| `zc workers` | List workers registered with the local broker |
| `zc update` | Update CLI from get.zakuro-ai.com |
| `zc pull` | Pull updated Docker images |
| `zc images` | List Zakuro images |
| `zc ps` | List running containers |
| `zc kill` | Stop running containers |
| `zc restart` | Restart containers |
## Environment Variables
| `ZAKURO_API_KEY` | Yes | — | Authentication token (also used for dashboard API calls in API mode) |
| `ZAKURO_MASTER_KEY` | No | — | Master API key for admin operations |
| `ZAKURO_API_URL` | No | — | Dashboard API URL (e.g., `https://my.zakuro-ai.com`) — when set together with `ZAKURO_API_KEY`, enables API mode (no direct PostgreSQL access) |
| `DATABASE_URL` | No | `postgresql://zakuro_broker:broker_secret_change_me@localhost:5432/zakuro` | PostgreSQL connection URL (restricted broker role) — ignored in API mode |
| `ZAKURO_WORKER_KEY` | No | — | Shared secret for worker management endpoints (`POST/DELETE /workers`, heartbeat) |
| `ZAKURO_TAILSCALE_IP` | No | — | Override Tailscale IP detection (Docker sidecar setups) |
| `TAILSCALE_AUTHKEY` | No | Auto-fetched | Tailscale auth key for joining the network mesh — automatically retrieved from dashboard API if `ZAKURO_API_URL` is set |
| `ZAKURO_PEERS` | No | — | Comma-separated `ip:port` peers for explicit discovery (workers) or `http://host:port` (brokers in P2P mode) |
| `ZAKURO_P2P` | No | `false` | Enable P2P broker-to-broker communication |
| `ZAKURO_PEER_KEY` | No | — | Shared secret for P2P broker authentication (`X-Peer-Key` header) |
| `ZAKURO_OWNER_ID` | No | — | This broker's unique ID for authority assignment (FNV-1a hash); auto-derived from `ZAKURO_API_KEY` if in `zk_{user}_{hex}` format |
| `ZAKURO_NODE_NAME` | No | Auto-detected | Human-readable node name (falls back to Tailscale hostname, then system hostname) |
| `ZAKURO_WORKER_PORT` | No | `3960` | Primary port workers listen on (discovery) |
| `ZAKURO_SCAN_RANGE` | No | — | Port range for discovery scan, e.g. `3960-3999` |
| `ZAKURO_SCAN_INTERVAL` | No | `15` | Seconds between discovery scans |
## Development
### Task Commands
This project uses [Task](https://taskfile.dev/) as its task runner. Install it with `go install github.com/go-task/task/v3/cmd/task@latest` or see [installation docs](https://taskfile.dev/installation/).
| `task build` | Build binary and start Docker services |
| `task build:bin` | Compile release binary and install to `/usr/local/bin/zc` |
| `task build:docker` | Build Docker image only |
| `task run` | Build and run the CLI |
| `task debug` | Build and run with `RUST_BACKTRACE=full` |
| `task release` | Run the installed release binary |
| `task broker` | Start broker in foreground |
| `task broker:tui` | Start broker with TUI dashboard |
| `task broker:daemon` | Start broker in daemon mode |
| `task bench` | Run default benchmark |
| `task bench:compare` | Compare all routing strategies |
| `task info` | Show system diagnostics |
| `task docker:up` | Start Docker services |
| `task docker:down` | Stop Docker services |
| `task docker:logs` | Tail Docker service logs |
| `task clean` | Remove build artifacts |
### Docker
```bash
# Build and run with Docker Compose
task build
# Or build the image directly
docker build -t zakuro-ai/broker -f docker/Dockerfile .
docker run -p 9000:9000 -e ZAKURO_API_KEY=demo zakuro-ai/broker
```
### Project Structure
```
src/
main.rs # CLI entry point and command routing
broker/
mod.rs # Broker state, config, and per-worker local detection
server.rs # HTTP server (tiny_http) with execute handler
worker.rs # Worker registry, resources, and health tracking
router.rs # Routing strategies and worker selection
credits.rs # Lightweight per-user balance cache and rate-limit tracker (reservation/history owned by Ledger)
ledger.rs # Central ledger: API mode (dashboard) or local in-memory reserve-commit
peer.rs # P2P broker-to-broker communication (PeerClient, PeerManager, FNV-1a authority)
flush.rs # TransactionBuffer for batched writes (5s cycle, via API or PG)
discovery.rs # Tailscale/local/peer-based worker discovery
wal.rs # Write-ahead log (JSONL + DashMap index, batched fsync)
recovery.rs # WAL replay on startup (reserved→cancel, executed→commit)
tui.rs # Interactive terminal dashboard (ratatui)
bench.rs # Benchmarking tool
stats.rs # Statistics tracking
info.rs # System diagnostics
common/mod.rs # Shared utilities
exec/mod.rs # Command execution helpers
manager/mod.rs # Docker container management
network/mod.rs # Network interface detection
envs/mod.rs # Environment variable handling
```
## Troubleshooting
- **"You are missing ZAKURO_API_KEY"** — set the `ZAKURO_API_KEY` environment variable before running any command.
- **No workers discovered** — ensure workers expose `/health` and `/info` endpoints on port 3960 (or `ZAKURO_WORKER_PORT`). Check with `zc info`.
- **PostgreSQL ledger unavailable** — the broker falls back to in-memory credits automatically. Set `DATABASE_URL` to connect with restricted `zakuro_broker` role.
- **P2P mode not working** — ensure `ZAKURO_P2P=true`, `ZAKURO_PEER_KEY` matches on all brokers, and `ZAKURO_PEERS` lists peer broker URLs.
- **TUI not rendering** — ensure your terminal supports 256 colors and is at least 80 columns wide.
## Documentation
See [docs/USAGE.md](docs/USAGE.md) for the complete usage guide covering all API details, configuration reference, and the credit system in depth.
See [docs/QUIC_TRANSPORT.md](docs/QUIC_TRANSPORT.md) for details on the QUIC-based P2P transport layer.
## License
MIT