zc2 0.0.5

Overview

zc is the Zakuro AI command-line tool that acts as a P2P compute broker. It routes compute requests to workers across a distributed network, handles credit-based billing via the Zakuro dashboard API with write-ahead logging, automatically discovers workers on Tailscale networks, and provides real-time monitoring through an interactive TUI dashboard.

Key features:

• P2P compute routing — forward requests to optimal workers based on price, latency, or availability
• P2P broker mesh — broker-to-broker communication with authoritative user assignment (ZAKURO_P2P mode)
• QUIC transport — high-throughput multiplexed UDP transport for peer task offers (QUIC-first with HTTP fallback)
• 6 routing strategies — best_price, best_latency, best_availability, round_robin, random, weighted_capacity
• Credit-based billing — dashboard API-backed ledger with atomic reserve-commit transactions; no direct database access
• Write-ahead log — JSONL WAL with DashMap in-memory index, batched fsync, and crash recovery
• Zero-latency hot path — in-memory DashMap credit operations for authoritative users (P2P mode)
• Automatic discovery — finds workers on Tailscale/WireGuard subnets or falls back to localhost
• Interactive TUI — real-time dashboard showing transactions, workers, metrics, and RPS sparklines
• Built-in benchmarking — measure throughput/latency and compare routing strategies
• Cluster management — Docker container lifecycle, image management, and diagnostics

Installation

Prerequisites

• Rust 1.75+
• Task (replaces Make)
• Docker and Docker Compose (optional, for containerized deployments)
• A Zakuro API key (ZAKURO_API_KEY) — sign up at zakuro-ai.com

From Source

git clone https://github.com/zakuro-ai/zc.git
cd zc
task build:bin

This compiles the release binary and installs it to /usr/local/bin/zc.

With Docker

task build

Builds the binary, then builds and starts the Docker services via docker compose.

Verify

zc --version

Quick Start

# 1. Set your API key
export ZAKURO_API_KEY="zk_myuser_abc123"

# 2. Set dashboard API URL
export ZAKURO_API_URL="https://my.zakuro-ai.com"

# 3. Start the broker with interactive TUI
zc -t broker

# The broker discovers workers, listens on 0.0.0.0:9000,
# and routes all credit operations through the dashboard API.
# Tailscale auth key is automatically retrieved from the dashboard.

Broker

The broker accepts compute requests, selects an optimal worker, forwards the request, and handles billing.

Modes

Mode	Command	Description
Foreground	zc broker	Live transaction log with colored output
TUI	zc -t broker	Interactive dashboard (workers, metrics, transactions)
Daemon	zc -d broker	Background process, minimal output

zc broker                # foreground on :9000
zc -t broker             # TUI on :9000
zc -d broker             # daemon on :9000
zc broker 8080           # custom port
zc broker 0.0.0.0 9000   # custom host and port

TUI Dashboard

The TUI (-t / --tui) provides four panels:

• Header — mode (P2P/LOCAL), ledger status, uptime, listen address
• Transactions — scrollable live log with status, user, cost, latency
• Workers — connected workers with resources, pricing, health
• Metrics — total requests, RPS sparkline, latency percentiles (p50/p90/p99), credits spent

Key	Action
Tab	Switch panels
j/k Up/Down	Scroll
g / G	Top / Bottom
?	Help overlay
q / Esc	Quit

Worker Discovery

Mode	Trigger	Behavior
Tailscale	tailscale0/wg0/ts* interface or ZAKURO_TAILSCALE_IP env	Scans peers + subnet; credits enforced for remote workers
Peers	ZAKURO_PEERS env set (no Tailscale interface)	Probes explicit peer addresses; credits enforced
Local	No Tailscale, no peers	Scans localhost:8089-8091; execution is free

Workers must expose /health and /info endpoints to be discovered. Workers can also register manually via the API.

In a P2P mesh, each broker detects its own Tailscale IP and treats workers at localhost or that IP as local (free execution). Remote workers are charged via the credit ledger. See docs/USAGE.md for mesh setup instructions.

P2P Broker-to-Broker Communication

Enable distributed broker mesh with ZAKURO_P2P=true. Each broker becomes authoritative for a subset of users:

Mode	Description
Authority::Local	User owned by this broker (via FNV-1a hash) → in-memory DashMap, zero PG on hot path
Authority::Peer(url)	User owned by remote broker → HTTP /peer/reserve + /peer/commit (~1ms RTT)
Authority::Standalone	Peer unreachable → dashboard API fallback path

Peer endpoints (X-Peer-Key auth):

• POST /peer/reserve — reserve credits for remote user
• POST /peer/commit — commit transaction after execution
• POST /peer/cancel — cancel reservation (refund)
• GET /peer/balance — get balance for remote user
• GET /peer/health — peer health check

Environment variables:

• ZAKURO_P2P=true — enable P2P mode (default: false)
• ZAKURO_PEER_KEY — shared secret for peer authentication
• ZAKURO_OWNER_ID — this broker's ID for authority assignment
• ZAKURO_PEERS — comma-separated peer broker URLs for discovery

Routing Strategies

Strategy	Description
best_price	Cheapest worker (default)
best_latency	Fastest response time
best_availability	Most available resources
round_robin	Even distribution
random	Random selection
weighted_capacity	Weighted by available CPU/memory

Set via the strategy field in X-Zakuro-Requirements header on /execute requests.

Credit System

• Dashboard API ledger — all balance and transaction operations go through ZAKURO_API_URL; no direct database access from the broker
• P2P mode (ZAKURO_P2P=true) — authoritative broker owns users via FNV-1a hash, DashMap in-memory operations on the hot path
• Write-ahead log — append-only JSONL with DashMap index (O(1) lookup), batched fsync (100ms/64 entries), crash recovery, auto-compaction every ~5 min
• Flush thread — batched writes every 5s (transactions + balance snapshots) in P2P mode, synced to the dashboard API
• Local mode — all execution is free when no Tailscale network is detected

# Check balance (requires Bearer token — own user or admin)
curl http://localhost:9000/credits/my-user \
  -H "Authorization: Bearer $API_KEY"

# Add credits (requires master API key)
curl -X POST http://localhost:9000/credits/my-user/add \
  -H "Content-Type: application/json" \
  -H "X-Api-Key: $ZAKURO_MASTER_KEY" \
  -d '{"amount": 100.0, "description": "Initial deposit"}'

Benchmarking

zc bench                          # default (10 concurrent, 1000 requests)
zc bench -c 100 -n 10000         # high-load
zc bench -s best_latency         # specific strategy
zc bench --compare               # compare all strategies
zc bench http://broker:9000      # remote broker

Option	Short	Default	Description
--concurrency	-c	10	Concurrent workers
--requests	-n	1000	Total requests
--url	-u	http://127.0.0.1:9000	Broker URL
--strategy	-s	best_price	Routing strategy
--compare			Compare all strategies
--list-strategies			List strategies

Output includes throughput (RPS), latency percentiles (p50-p99), success/failure rates, and an ASCII histogram.

Remote Monitoring

Attach to a running broker from another machine:

zc attach localhost:9000
zc attach http://broker.tailnet:9000

Renders the same TUI dashboard using the broker's /stats endpoint. Press r to force refresh.

System Diagnostics

zc info

Shows network info (hostname, mode, IPs), compute cluster availability (Ray, Dask, Spark), environment variables, and broker port status.

API Endpoints

The broker listens on 0.0.0.0:9000 by default.

Method	Endpoint	Auth	Description
GET	/health	None	Health check
GET	/stats	Bearer	Broker statistics (non-admin: own txns only)
GET	/workers	None	List registered workers
POST	/workers	X-Worker-Key	Register a worker
POST	/workers/heartbeat	X-Worker-Key	Worker heartbeat
DELETE	/workers/:id	X-Worker-Key	Unregister worker
POST	/execute	Bearer	Execute compute request
POST	/price	None	Estimate request price
GET	/credits/:user	Bearer	Get balance (own user or admin)
POST	/credits/:user/add	X-Api-Key (master)	Add credits
GET	/me	Bearer	Current user info
GET	/ledger/status	Bearer (admin)	Ledger status (API URL, connection state)
P2P Endpoints (ZAKURO_P2P=true)
POST	/peer/reserve	X-Peer-Key	Reserve credits for remote user
POST	/peer/commit	X-Peer-Key	Commit transaction after execution
POST	/peer/cancel	X-Peer-Key	Cancel reservation (refund)
POST	/peer/earn	X-Peer-Key	Credit earnings to broker owner
GET	/peer/balance	X-Peer-Key	Get balance for remote user
GET	/peer/workers	X-Peer-Key	List this broker's local workers
GET	/peer/health	X-Peer-Key	Peer broker health check

Worker Registration

{
  "name": "worker-1",
  "uri": "http://10.13.13.5:8089",
  "worker_type": "zakuro",
  "resources": {
    "cpus_total": 8.0,
    "cpus_available": 8.0,
    "memory_total": 17179869184,
    "memory_available": 17179869184,
    "gpus_total": 1,
    "gpus_available": 1
  },
  "pricing": {
    "cpu_price": 0.001,
    "memory_price": 0.0001,
    "gpu_price": 0.01,
    "min_charge": 0.001
  }
}

Cluster Commands

Command	Description
zc up	Start 1 worker + broker locally (foreground)
zc up --workers N	Start N workers + broker
zc down	Stop all workers + broker started by zc up
zc workers	List workers registered with the local broker
zc update	Update CLI from get.zakuro-ai.com
zc pull	Pull updated Docker images
zc images	List Zakuro images
zc ps	List running containers
zc kill	Stop running containers
zc restart	Restart containers

Environment Variables

Variable	Required	Default	Description
ZAKURO_API_KEY	Yes	—	Authentication token (also used for dashboard API calls in API mode)
ZAKURO_MASTER_KEY	No	—	Master API key for admin operations
ZAKURO_API_URL	Yes	—	Dashboard API URL (e.g., https://my.zakuro-ai.com) — required for credit operations and worker sync
ZAKURO_WORKER_KEY	No	—	Shared secret for worker management endpoints (POST/DELETE /workers, heartbeat)
ZAKURO_TAILSCALE_IP	No	—	Override Tailscale IP detection (Docker sidecar setups)
TAILSCALE_AUTHKEY	No	Auto-fetched	Tailscale auth key for joining the network mesh — automatically retrieved from dashboard API if ZAKURO_API_URL is set
ZAKURO_PEERS	No	—	Comma-separated ip:port peers for explicit discovery (workers) or http://host:port (brokers in P2P mode)
ZAKURO_P2P	No	false	Enable P2P broker-to-broker communication
ZAKURO_PEER_KEY	No	—	Shared secret for P2P broker authentication (X-Peer-Key header)
ZAKURO_OWNER_ID	No	—	This broker's unique ID for authority assignment (FNV-1a hash); auto-derived from ZAKURO_API_KEY if in zk_{user}_{hex} format
ZAKURO_NODE_NAME	No	Auto-detected	Human-readable node name (falls back to Tailscale hostname, then system hostname)
ZAKURO_WORKER_PORT	No	3960	Primary port workers listen on (discovery)
ZAKURO_SCAN_RANGE	No	—	Port range for discovery scan, e.g. 3960-3999
ZAKURO_SCAN_INTERVAL	No	15	Seconds between discovery scans

Development

Task Commands

This project uses Task as its task runner. Install it with go install github.com/go-task/task/v3/cmd/task@latest or see installation docs.

Command	Description
task build	Build binary and start Docker services
task build:bin	Compile release binary and install to /usr/local/bin/zc
task build:docker	Build Docker image only
task run	Build and run the CLI
task debug	Build and run with RUST_BACKTRACE=full
task release	Run the installed release binary
task broker	Start broker in foreground
task broker:tui	Start broker with TUI dashboard
task broker:daemon	Start broker in daemon mode
task bench	Run default benchmark
task bench:compare	Compare all routing strategies
task info	Show system diagnostics
task docker:up	Start Docker services
task docker:down	Stop Docker services
task docker:logs	Tail Docker service logs
task clean	Remove build artifacts

Docker

# Build and run with Docker Compose
task build

# Or build the image directly
docker build -t zakuro-ai/broker -f docker/Dockerfile .
docker run -p 9000:9000 -e ZAKURO_API_KEY=demo zakuro-ai/broker

Project Structure

src/
  main.rs              # CLI entry point and command routing
  broker/
    mod.rs             # Broker state, config, and per-worker local detection
    server.rs          # HTTP server (tiny_http) with execute handler
    worker.rs          # Worker registry, resources, and health tracking
    router.rs          # Routing strategies and worker selection
    credits.rs         # Lightweight per-user balance cache and rate-limit tracker (reservation/history owned by Ledger)
    ledger.rs          # Central ledger: API mode (dashboard) or local in-memory reserve-commit
    peer.rs            # P2P broker-to-broker communication (PeerClient, PeerManager, FNV-1a authority)
    flush.rs           # TransactionBuffer for batched writes (5s cycle, via dashboard API)
    discovery.rs       # Tailscale/local/peer-based worker discovery
    wal.rs             # Write-ahead log (JSONL + DashMap index, batched fsync)
    recovery.rs        # WAL replay on startup (reserved→cancel, executed→commit)
    tui.rs             # Interactive terminal dashboard (ratatui)
    bench.rs           # Benchmarking tool
    stats.rs           # Statistics tracking
    info.rs            # System diagnostics
  common/mod.rs        # Shared utilities
  exec/mod.rs          # Command execution helpers
  manager/mod.rs       # Docker container management
  network/mod.rs       # Network interface detection
  envs/mod.rs          # Environment variable handling

Troubleshooting

• "You are missing ZAKURO_API_KEY" — set the ZAKURO_API_KEY environment variable before running any command.
• No workers discovered — ensure workers expose /health and /info endpoints on port 3960 (or ZAKURO_WORKER_PORT). Check with zc info.
• Credit operations failing — ensure both ZAKURO_API_KEY and ZAKURO_API_URL are set. The broker requires the dashboard API for all billing operations.
• P2P mode not working — ensure ZAKURO_P2P=true, ZAKURO_PEER_KEY matches on all brokers, and ZAKURO_PEERS lists peer broker URLs.
• TUI not rendering — ensure your terminal supports 256 colors and is at least 80 columns wide.

Documentation

See docs/USAGE.md for the complete usage guide covering all API details, configuration reference, and the credit system in depth.

See docs/QUIC_TRANSPORT.md for details on the QUIC-based P2P transport layer.

License

MIT