zc2 0.0.28

P2P compute broker with credit-based billing, WAL, and broker mesh support
zc2-0.0.28 is not a library.

zakuro Logo

Overview

zc is the Zakuro AI command-line tool that acts as a P2P compute broker. It routes compute requests to workers across a distributed network, handles credit-based billing via the Zakuro dashboard API with write-ahead logging, automatically discovers workers on the WireGuard mesh, and reports live transactions, workers and metrics from the CLI and the broker's /stats endpoint.

Key features:

  • P2P compute routing — forward requests to optimal workers based on price, latency, or availability
  • P2P broker mesh — broker-to-broker communication with authoritative user assignment (ZAKURO_P2P mode)
  • QUIC transport — high-throughput multiplexed UDP transport for peer task offers (QUIC-first with HTTP fallback)
  • 6 routing strategies — best_price, best_latency, best_availability, round_robin, random, weighted_capacity
  • Credit-based billing — dashboard API-backed ledger with atomic reserve-commit transactions; no direct database access; standalone mode (ZAKURO_MASTER_KEY) runs a local ledger without dashboard
  • Write-ahead log — JSONL WAL with DashMap in-memory index, batched fsync, and crash recovery
  • Zero-latency hot path — in-memory DashMap credit operations for authoritative users (P2P mode)
  • Automatic discovery — finds workers on the mesh subnet (parallel scan) or falls back to localhost
  • Built-in benchmarking — measure throughput/latency and compare routing strategies
  • Cluster management — Docker container lifecycle, image management, and diagnostics
  • Zakuro Drive hooks — manage file-event hooks (webhook/exec) on a local zakuro-drive daemon over its QUIC API

Installation

Prerequisites

  • Rust — recent stable (the dependency tree's MSRV is 1.88 via time 0.3.47; the old "1.75+" note predates that)
  • Task (replaces Make)
  • Docker and Docker Compose (optional, for containerized deployments)
  • A Zakuro API key (ZAKURO_API_KEY) — sign up at zakuro-ai.com

From Source

git clone https://github.com/zakuro-ai/zc.git
cd zc
task build:bin

This compiles the release binary and installs it to /usr/local/bin/zc.

Install with Cargo (no clone)

cargo install --git https://github.com/zakuro-ai/zc zc2

zc2 needs no access to any private repository — it simply does not include zc hooks (Zakuro Drive file-event hooks). That command lives in a separate zc-hooks helper binary which zc hooks execs, because it depends on the private zakuro-client crate. Building it requires read access to zakuro-ai/zakuro-drive:

cargo build --release -p zc-hooks

Put the resulting zc-hooks next to your zc binary (or anywhere on PATH) and zc hooks … works as before.

From crates.io

cargo install zc2

Add --locked to get the exact dependency tree CI tests; without it cargo resolves the newest compatible versions.

Note the registry build has no zc hooks — that command needs the zc-hooks sidecar, which depends on a private crate and so cannot be published. Use a binary release if you need it.

With Docker

task build

Builds the binary, then builds and starts the Docker services via docker compose.

Verify

zc --version

Quick Start

# 1. Set your API key
export ZAKURO_API_KEY="zk_myuser_abc123"

# 2. Set dashboard API URL
export ZAKURO_API_URL="https://my.zakuro-ai.com"

# 3. Start the broker
zc broker

# The broker discovers workers, listens on 0.0.0.0:9000,
# and routes all credit operations through the dashboard API.

Zakuro Drive Hooks

zc hooks manages file-event hooks on a running zakuro-drive daemon (zakurod) over its QUIC API. A hook fires on file events (file.created | file.modified | file.deleted) matching a glob, and either POSTs a signed JSON payload to a webhook or runs a local command.

# Webhook hook — secret read from stdin (recommended; avoids ps/history leaks)
echo "$WEBHOOK_SECRET" | zc hooks add --name slack-notify \
  --on file.created --on file.modified --glob "reports/**/*.pdf" \
  --webhook https://hooks.example.com/zakuro --secret-stdin

# Local command hook with a custom timeout
zc hooks add --name csv-to-parquet --on file.created --glob "**/*.csv" \
  --exec "/usr/local/bin/convert-csv" --timeout 120

zc hooks list --all          # include disabled hooks
zc hooks get csv-to-parquet  # full config (secrets redacted by the daemon)
zc hooks update csv-to-parquet --disable
zc hooks logs --failed --from 2h
zc hooks remove csv-to-parquet

The daemon address defaults to 127.0.0.1:4719; override with --daemon HOST:PORT or ZAKURO_DRIVE_DAEMON. Run zc hooks --help for the full flag reference (retry tuning, --secret-from-env, --tls-insecure, dead-letter views).

Packaging note: zakuro-client (the drive daemon API client) is consumed as a git dependency on the private zakuro-ai/zakuro-drive repository, pinned to the client-v0.1.0 tag. crates.io rejects git dependencies, so the crates.io publish job in auto-release.yml is paused until zakuro-proto + zakuro-client are published to a registry; GitHub releases with prebuilt binaries continue.

Broker

The broker accepts compute requests, selects an optimal worker, forwards the request, and handles billing.

Modes

Mode Command Description
Foreground zc broker Live transaction log with colored output
Daemon zc -d broker Background process, minimal output 
zc broker                # foreground on :9000
zc -d broker             # daemon on :9000
zc broker 8080           # custom port
zc broker 0.0.0.0 9000   # custom host and port

Worker Discovery

Mode Trigger Behavior
Mesh a zakuro0 or wg* interface with a 10.13.13.x address, or ZAKURO_WIREGUARD_IP env Scans peers + subnet; credits enforced for remote workers
Peers ZAKURO_PEERS env set (no mesh interface) Probes explicit peer addresses; credits enforced
Local No mesh, no peers Scans localhost:8089-8091; execution is free

Workers must expose /health and /info endpoints to be discovered. Workers can also register manually via the API.

In a P2P mesh, each broker detects its own mesh IP and treats workers at localhost/127.0.0.1 or at that IP as local (free execution). Remote workers (mesh peers) are charged via the credit ledger. See docs/USAGE.md for mesh setup instructions.

Subnet scanning is parallelized: phase 1 does a concurrent TCP SYN to all 253 /24 hosts (50ms timeout); phase 2 probes only reachable hosts. This reduces scan time from 253 × 50ms to ~50ms + probe time.

P2P Broker-to-Broker Communication

Enable distributed broker mesh with ZAKURO_P2P=true. Each broker becomes authoritative for a subset of users:

Mode Description
Authority::Local User owned by this broker (via FNV-1a hash) → in-memory DashMap, zero PG on hot path
Authority::Peer(url) User owned by remote broker → HTTP /peer/reserve + /peer/commit (~1ms RTT)
Authority::Standalone Peer unreachable → dashboard API fallback path

Peer endpoints (X-Peer-Key auth):

  • POST /peer/reserve — reserve credits for remote user
  • POST /peer/commit — commit transaction after execution
  • POST /peer/cancel — cancel reservation (refund)
  • GET /peer/balance — get balance for remote user
  • GET /peer/health — peer health check

Environment variables:

  • ZAKURO_P2P=true — enable P2P mode (default: false)
  • ZAKURO_PEER_KEY — shared secret for peer authentication
  • ZAKURO_OWNER_ID — this broker's ID for authority assignment
  • ZAKURO_PEERS — comma-separated peer broker URLs for discovery

Routing Strategies

Strategy Description
best_price Cheapest worker (default)
best_latency Fastest response time
best_availability Most available resources
round_robin Even distribution
random Random selection
weighted_capacity Weighted by available CPU/memory

Set via the strategy field in X-Zakuro-Requirements header on /execute requests.

Credit System

  • Dashboard API ledger — all balance and transaction operations go through ZAKURO_API_URL; no direct database access from the broker. The dashboard returns balance in the credits_balance field.
  • Standalone mode — set ZAKURO_MASTER_KEY (no ZAKURO_API_URL needed) to run a fully local credit ledger. Transactions are queued in the WAL and the flush buffer; billing is enforced even without the dashboard.
  • P2P mode (ZAKURO_P2P=true) — authoritative broker owns users via FNV-1a hash, DashMap in-memory operations on the hot path. Setting remote_only: true in a request forces routing to peer workers (bypasses local worker selection).
  • Write-ahead log — append-only JSONL with DashMap index (O(1) lookup), batched fsync (100ms/64 entries), crash recovery, auto-compaction every ~5 min
  • Flush thread — batched writes every 5s (transactions + balance snapshots), synced to the dashboard API
  • Local mode — all execution is free when no mesh network is detected
# Check balance (requires Bearer token — own user or admin)
curl http://localhost:9000/credits/my-user \
  -H "Authorization: Bearer $API_KEY"

# Add credits (requires master API key)
curl -X POST http://localhost:9000/credits/my-user/add \
  -H "Content-Type: application/json" \
  -H "X-Api-Key: $ZAKURO_MASTER_KEY" \
  -d '{"amount": 100.0, "description": "Initial deposit"}'

Benchmarking

zc bench                          # default (10 concurrent, 1000 requests)
zc bench -c 100 -n 10000         # high-load
zc bench -s best_latency         # specific strategy
zc bench --compare               # compare all strategies
zc bench http://broker:9000      # remote broker
zc bench mesh --workers a:3960,b:3960,c:3960 | jq .
Option Short Default Description
--concurrency -c 10 Concurrent workers
--requests -n 1000 Total requests
--url -u http://127.0.0.1:9000 Broker URL
--strategy -s best_price Routing strategy
--compare Compare all strategies
--list-strategies List strategies

Output includes throughput (RPS), latency percentiles (p50-p99), success/failure rates, and an ASCII histogram.

zc bench mesh emits a JSON mesh-warmup report with per-worker latency_p95, bandwidth_bps, and recommended_backpressure, suitable for jq or for feeding into operator tooling.

Remote Monitoring

Read a running broker from another machine:

zc workers localhost:9000            # its workers, their health and prices
zc workers http://broker.mesh:9000
# transactions, request rate and latency percentiles, as JSON
curl -s -H "Authorization: Bearer $ZAKURO_API_KEY" http://broker.mesh:9000/stats

System Diagnostics

zc info

Shows network info (hostname, mode, IPs), compute cluster availability (Ray, Dask, Spark), environment variables, and broker port status.

API Endpoints

The broker listens on 0.0.0.0:9000 by default.

Method Endpoint Auth Description
GET /health None Health check
GET /stats Bearer Broker statistics (non-admin: own txns only)
GET /workers None List registered workers
POST /workers X-Worker-Key Register a worker
POST /workers/heartbeat X-Worker-Key Worker heartbeat
DELETE /workers/:id X-Worker-Key Unregister worker
POST /execute Bearer Execute compute request
POST /price None Estimate request price
GET /credits/:user Bearer Get balance (own user or admin)
POST /credits/:user/add X-Api-Key (master) Add credits
GET /me Bearer Current user info
GET /ledger/status Bearer (admin) Ledger status (API URL, connection state)
P2P Endpoints (ZAKURO_P2P=true)
POST /peer/reserve X-Peer-Key Reserve credits for remote user
POST /peer/commit X-Peer-Key Commit transaction after execution
POST /peer/cancel X-Peer-Key Cancel reservation (refund)
POST /peer/earn X-Peer-Key Credit earnings to broker owner
GET /peer/balance X-Peer-Key Get balance for remote user
GET /peer/workers X-Peer-Key List this broker's local workers
GET /peer/health X-Peer-Key Peer broker health check

Execute Request Options

The /execute endpoint accepts a JSON body (or X-Zakuro-Requirements header) with:

Field Type Default Description
cpus float 1.0 Required vCPUs
memory_bytes int — Required memory in bytes
gpus int 0 Required GPUs
estimated_duration_secs float 1.0 Estimated job duration (used for cost estimation)
budget_credits float — Maximum credits to spend; request fails if cost would exceed this
strategy string best_price Routing strategy
remote_only bool false When true, skip local workers and dispatch only to peer brokers (requires ZAKURO_P2P=true)
worker_type string — Filter workers by type (e.g. standard, premium)

Response headers include X-Zakuro-Worker (worker ID), X-Zakuro-Cost (credits spent), and X-Zakuro-Credits-Remaining (balance after execution).

Worker Registration

{
  "name": "worker-1",
  "uri": "http://10.13.13.5:8089",
  "worker_type": "zakuro",
  "resources": {
    "cpus_total": 8.0,
    "cpus_available": 8.0,
    "memory_total": 17179869184,
    "memory_available": 17179869184,
    "gpus_total": 1,
    "gpus_available": 1
  },
  "pricing": {
    "cpu_price": 0.001,
    "memory_price": 0.0001,
    "gpu_price": 0.01,
    "min_charge": 0.001
  }
}

Cluster Commands

Command Description
zc up Start 1 worker + broker locally (foreground)
zc up --workers N Start N workers + broker
zc down Stop all workers + broker started by zc up
zc workers List workers registered with the local broker
zc update Update CLI from get.zakuro-ai.com
zc pull Pull updated Docker images
zc images List Zakuro images
zc ps List running containers
zc kill Stop running containers
zc restart Restart containers

Environment Variables

Variable Required Default Description
ZAKURO_API_KEY No* — Authentication token for dashboard API. *Optional when using only local or discovered brokers (P2P only).
ZAKURO_MASTER_KEY No — Master API key for admin operations and standalone billing. When set (without ZAKURO_API_URL), the broker runs a fully local credit ledger with billing enforced.
ZAKURO_API_URL No* — Dashboard API URL (e.g., https://my.zakuro-ai.com). *Required for dashboard-backed billing and worker sync. Omit to use standalone mode (ZAKURO_MASTER_KEY).
ZAKURO_WORKER_KEY No — Shared secret for worker management endpoints (POST/DELETE /workers, heartbeat)
ZAKURO_WIREGUARD_IP No — Override mesh IP detection (Docker sidecar setups). Named for the mesh we used to run; it applies to WireGuard too.
ZAKURO_PEERS No — Comma-separated ip:port peers for explicit discovery (workers) or http://host:port (brokers in P2P mode)
ZAKURO_DISCOVER_BROKER_PEERS No true When P2P is enabled and ZAKURO_PEERS is empty, scan localhost ports 9000–9010 for other brokers. Set to false to disable. When using discovered brokers, ZAKURO_API_KEY is optional.
ZAKURO_P2P No false Enable P2P broker-to-broker communication
ZAKURO_PEER_KEY No — Shared secret for P2P broker authentication (X-Peer-Key header)
ZAKURO_OWNER_ID — — Not configurable. Always derived from ZAKURO_API_KEY (zk_{user_id}_{hex} format). Manual overrides are ignored.
ZAKURO_NODE_NAME No Auto-detected Human-readable node name (falls back to the mesh hostname, then system hostname)
ZAKURO_WORKER_PORT No 3960 Primary port workers listen on (discovery)
ZAKURO_SCAN_RANGE No — Port range for discovery scan, e.g. 3960-3999
ZAKURO_SCAN_INTERVAL No 15 Seconds between discovery scans

Development

Task Commands

This project uses Task as its task runner. Install it with go install github.com/go-task/task/v3/cmd/task@latest or see installation docs.

Command Description
task build Build binary and start Docker services
task build:bin Incremental release build of zc and zc-hooks, installed to ~/.local/bin
task dev Incremental debug build for fixing bugs, linked as ~/.local/bin/zc-dev
task build:docker Build Docker image only
task run Build and run the CLI
task debug Build and run with RUST_BACKTRACE=full
task release Run the installed release binary
task broker Start broker in foreground
task broker:daemon Start broker in daemon mode
task bench Run default benchmark
task bench:compare Compare all routing strategies
task info Show system diagnostics
task docker:up Start Docker services
task docker:down Stop Docker services
task docker:logs Tail Docker service logs
task clean Remove build artifacts

Docker

# Build and run with Docker Compose
task build

# Or build the image directly
docker build -t zakuro-ai/broker -f docker/Dockerfile .
docker run -p 9000:9000 -e ZAKURO_API_KEY=demo zakuro-ai/broker

Project Structure

src/
  main.rs              # CLI entry point and command routing
  broker/
    mod.rs             # Broker state, config, and per-worker local detection
    server.rs          # HTTP server (tiny_http) with execute handler
    worker.rs          # Worker registry, resources, and health tracking
    router.rs          # Routing strategies and worker selection
    credits.rs         # Lightweight per-user balance cache and rate-limit tracker (reservation/history owned by Ledger)
    ledger.rs          # Central ledger: API mode (dashboard) or local in-memory reserve-commit
    peer.rs            # P2P broker-to-broker communication (PeerClient, PeerManager, FNV-1a authority)
    flush.rs           # TransactionBuffer for batched writes (5s cycle, via dashboard API)
    discovery.rs       # mesh/local/peer-based worker discovery
    wal.rs             # Write-ahead log (JSONL + DashMap index, batched fsync)
    recovery.rs        # WAL replay on startup (reserved→cancel, executed→commit)
    bench.rs           # Benchmarking tool
    stats.rs           # Statistics tracking
    info.rs            # System diagnostics
  common/mod.rs        # Shared utilities
  exec/mod.rs          # Command execution helpers
  manager/mod.rs       # Docker container management
  network/mod.rs       # Network interface detection
  envs/mod.rs          # Environment variable handling

Troubleshooting

  • "You are missing ZAKURO_API_KEY" — set ZAKURO_API_KEY for dashboard auth. For standalone billing, set ZAKURO_MASTER_KEY instead. P2P-only setups (auto-discovered peers) don't require either.
  • No workers discovered — ensure workers expose /health and /info endpoints on port 3960 (or ZAKURO_WORKER_PORT). Check with zc info.
  • Credit operations failing — ensure both ZAKURO_API_KEY and ZAKURO_API_URL are set for dashboard mode. For standalone mode, set only ZAKURO_MASTER_KEY. Verify the dashboard returns the credits_balance field in /api/broker/balance/{id}.
  • Zero cost on remote executions — ensure is_billing_enabled() returns true: either dashboard URL+key or ZAKURO_MASTER_KEY must be set. Local workers (127.0.0.1/localhost) are always free; remote mesh workers are billed.
  • remote_only: true returns 503 — ensure ZAKURO_P2P=true and at least one peer broker is reachable. Without peers, remote_only requests cannot be served.
  • P2P mode not working — ensure ZAKURO_P2P=true, ZAKURO_PEER_KEY matches on all brokers. Use ZAKURO_PEERS for explicit URLs, or leave empty to auto-discover brokers on localhost (ports 9000–9010).
  • Mesh IP not detected — the broker recognises a zakuro0 or wg* interface holding an address in 10.13.13.0/24 (see get_mesh_ip in src/broker/discovery.rs). An interface outside that subnet is ignored. Set ZAKURO_WIREGUARD_IP to override.

Documentation

See docs/USAGE.md for the complete usage guide covering all API details, configuration reference, and the credit system in depth.

See docs/QUIC_TRANSPORT.md for details on the QUIC-based P2P transport layer.

License

MIT