zc2 0.0.25

Overview

zc is the Zakuro AI command-line tool that acts as a P2P compute broker. It routes compute requests to workers across a distributed network, handles credit-based billing via the Zakuro dashboard API with write-ahead logging, automatically discovers workers on the WireGuard mesh, and provides real-time monitoring through an interactive TUI dashboard.

Key features:

• P2P compute routing — forward requests to optimal workers based on price, latency, or availability
• P2P broker mesh — broker-to-broker communication with authoritative user assignment (ZAKURO_P2P mode)
• QUIC transport — high-throughput multiplexed UDP transport for peer task offers (QUIC-first with HTTP fallback)
• 6 routing strategies — best_price, best_latency, best_availability, round_robin, random, weighted_capacity
• Credit-based billing — dashboard API-backed ledger with atomic reserve-commit transactions; no direct database access; standalone mode (ZAKURO_MASTER_KEY) runs a local ledger without dashboard
• Write-ahead log — JSONL WAL with DashMap in-memory index, batched fsync, and crash recovery
• Zero-latency hot path — in-memory DashMap credit operations for authoritative users (P2P mode)
• Automatic discovery — finds workers on the mesh subnet (parallel scan) or falls back to localhost
• Interactive TUI — real-time dashboard showing transactions, workers, metrics, and RPS sparklines
• Built-in benchmarking — measure throughput/latency and compare routing strategies
• Cluster management — Docker container lifecycle, image management, and diagnostics
• Zakuro Drive hooks — manage file-event hooks (webhook/exec) on a local zakuro-drive daemon over its QUIC API

Installation

Prerequisites

• Rust — recent stable (the dependency tree's MSRV is 1.88 via time 0.3.47; the old "1.75+" note predates that)
• Task (replaces Make)
• Docker and Docker Compose (optional, for containerized deployments)
• A Zakuro API key (ZAKURO_API_KEY) — sign up at zakuro-ai.com

From Source

git clone https://github.com/zakuro-ai/zc.git
cd zc
task build:bin

This compiles the release binary and installs it to /usr/local/bin/zc.

Install with Cargo (no clone)

cargo install --git https://github.com/zakuro-ai/zc zc2

zc2 needs no access to any private repository — it simply does not include zc hooks (Zakuro Drive file-event hooks). That command lives in a separate zc-hooks helper binary which zc hooks execs, because it depends on the private zakuro-client crate. Building it requires read access to zakuro-ai/zakuro-drive:

cargo build --release -p zc-hooks

Put the resulting zc-hooks next to your zc binary (or anywhere on PATH) and zc hooks … works as before.

From crates.io

cargo install zc2 --locked

--locked is required, not optional. tui-textarea 0.4 declares ratatui >=0.23.0, <1, so a fresh dependency resolution satisfies it with ratatui 0.30 while zc2 itself pins ratatui 0.26 — two incompatible copies, and the build fails with expected ratatui_core::style::Style, found ratatui::style::Style. --locked uses the lockfile shipped in the published crate, which pins a single ratatui 0.26.3. Tracked in #196.

Note the registry build has no zc hooks — that command needs the zc-hooks sidecar, which depends on a private crate and so cannot be published. Use a binary release if you need it.

With Docker

task build

Builds the binary, then builds and starts the Docker services via docker compose.

Verify

zc --version

Quick Start

# 1. Set your API key
export ZAKURO_API_KEY="zk_myuser_abc123"

# 2. Set dashboard API URL
export ZAKURO_API_URL="https://my.zakuro-ai.com"

# 3. Start the broker with interactive TUI
zc -t broker

# The broker discovers workers, listens on 0.0.0.0:9000,
# and routes all credit operations through the dashboard API.

Zakuro Drive Hooks

zc hooks manages file-event hooks on a running zakuro-drive daemon (zakurod) over its QUIC API. A hook fires on file events (file.created | file.modified | file.deleted) matching a glob, and either POSTs a signed JSON payload to a webhook or runs a local command.

# Webhook hook — secret read from stdin (recommended; avoids ps/history leaks)
echo "$WEBHOOK_SECRET" | zc hooks add --name slack-notify \
  --on file.created --on file.modified --glob "reports/**/*.pdf" \
  --webhook https://hooks.example.com/zakuro --secret-stdin

# Local command hook with a custom timeout
zc hooks add --name csv-to-parquet --on file.created --glob "**/*.csv" \
  --exec "/usr/local/bin/convert-csv" --timeout 120

zc hooks list --all          # include disabled hooks
zc hooks get csv-to-parquet  # full config (secrets redacted by the daemon)
zc hooks update csv-to-parquet --disable
zc hooks logs --failed --from 2h
zc hooks remove csv-to-parquet

The daemon address defaults to 127.0.0.1:4719; override with --daemon HOST:PORT or ZAKURO_DRIVE_DAEMON. Run zc hooks --help for the full flag reference (retry tuning, --secret-from-env, --tls-insecure, dead-letter views).

Packaging note: zakuro-client (the drive daemon API client) is consumed as a git dependency on the private zakuro-ai/zakuro-drive repository, pinned to the client-v0.1.0 tag. crates.io rejects git dependencies, so the crates.io publish job in auto-release.yml is paused until zakuro-proto + zakuro-client are published to a registry; GitHub releases with prebuilt binaries continue.

Broker

The broker accepts compute requests, selects an optimal worker, forwards the request, and handles billing.

Modes

Mode	Command	Description
Foreground	zc broker	Live transaction log with colored output
TUI	zc -t broker	Interactive dashboard (workers, metrics, transactions)
Daemon	zc -d broker	Background process, minimal output

zc broker                # foreground on :9000
zc -t broker             # TUI on :9000
zc -d broker             # daemon on :9000
zc broker 8080           # custom port
zc broker 0.0.0.0 9000   # custom host and port

TUI Dashboard

The TUI (-t / --tui) provides four panels:

• Header — mode (P2P/LOCAL), ledger status, uptime, listen address
• Transactions — scrollable live log with status, user, cost, latency
• Workers — connected workers with resources, pricing, health
• Metrics — total requests, RPS sparkline, latency percentiles (p50/p90/p99), credits spent

Key	Action
Tab	Switch panels
j/k Up/Down	Scroll
g / G	Top / Bottom
?	Help overlay
q / Esc	Quit

Worker Discovery

Mode	Trigger	Behavior
Mesh	a zakuro0 or wg* interface with a 10.13.13.x address, or ZAKURO_WIREGUARD_IP env	Scans peers + subnet; credits enforced for remote workers
Peers	ZAKURO_PEERS env set (no mesh interface)	Probes explicit peer addresses; credits enforced
Local	No mesh, no peers	Scans localhost:8089-8091; execution is free

Workers must expose /health and /info endpoints to be discovered. Workers can also register manually via the API.

In a P2P mesh, each broker detects its own mesh IP and treats workers at localhost/127.0.0.1 or at that IP as local (free execution). Remote workers (mesh peers) are charged via the credit ledger. See docs/USAGE.md for mesh setup instructions.

Subnet scanning is parallelized: phase 1 does a concurrent TCP SYN to all 253 /24 hosts (50ms timeout); phase 2 probes only reachable hosts. This reduces scan time from 253 × 50ms to ~50ms + probe time.

P2P Broker-to-Broker Communication

Enable distributed broker mesh with ZAKURO_P2P=true. Each broker becomes authoritative for a subset of users:

Mode	Description
Authority::Local	User owned by this broker (via FNV-1a hash) → in-memory DashMap, zero PG on hot path
Authority::Peer(url)	User owned by remote broker → HTTP /peer/reserve + /peer/commit (~1ms RTT)
Authority::Standalone	Peer unreachable → dashboard API fallback path

Peer endpoints (X-Peer-Key auth):

• POST /peer/reserve — reserve credits for remote user
• POST /peer/commit — commit transaction after execution
• POST /peer/cancel — cancel reservation (refund)
• GET /peer/balance — get balance for remote user
• GET /peer/health — peer health check

Environment variables:

• ZAKURO_P2P=true — enable P2P mode (default: false)
• ZAKURO_PEER_KEY — shared secret for peer authentication
• ZAKURO_OWNER_ID — this broker's ID for authority assignment
• ZAKURO_PEERS — comma-separated peer broker URLs for discovery

Routing Strategies

Strategy	Description
best_price	Cheapest worker (default)
best_latency	Fastest response time
best_availability	Most available resources
round_robin	Even distribution
random	Random selection
weighted_capacity	Weighted by available CPU/memory

Set via the strategy field in X-Zakuro-Requirements header on /execute requests.

Credit System

• Dashboard API ledger — all balance and transaction operations go through ZAKURO_API_URL; no direct database access from the broker. The dashboard returns balance in the credits_balance field.
• Standalone mode — set ZAKURO_MASTER_KEY (no ZAKURO_API_URL needed) to run a fully local credit ledger. Transactions are queued in the WAL and the flush buffer; billing is enforced even without the dashboard.
• P2P mode (ZAKURO_P2P=true) — authoritative broker owns users via FNV-1a hash, DashMap in-memory operations on the hot path. Setting remote_only: true in a request forces routing to peer workers (bypasses local worker selection).
• Write-ahead log — append-only JSONL with DashMap index (O(1) lookup), batched fsync (100ms/64 entries), crash recovery, auto-compaction every ~5 min
• Flush thread — batched writes every 5s (transactions + balance snapshots), synced to the dashboard API
• Local mode — all execution is free when no mesh network is detected

# Check balance (requires Bearer token — own user or admin)
curl http://localhost:9000/credits/my-user \
  -H "Authorization: Bearer $API_KEY"

# Add credits (requires master API key)
curl -X POST http://localhost:9000/credits/my-user/add \
  -H "Content-Type: application/json" \
  -H "X-Api-Key: $ZAKURO_MASTER_KEY" \
  -d '{"amount": 100.0, "description": "Initial deposit"}'

Benchmarking

zc bench                          # default (10 concurrent, 1000 requests)
zc bench -c 100 -n 10000         # high-load
zc bench -s best_latency         # specific strategy
zc bench --compare               # compare all strategies
zc bench http://broker:9000      # remote broker
zc bench mesh --workers a:3960,b:3960,c:3960 | jq .

Option	Short	Default	Description
--concurrency	-c	10	Concurrent workers
--requests	-n	1000	Total requests
--url	-u	http://127.0.0.1:9000	Broker URL
--strategy	-s	best_price	Routing strategy
--compare			Compare all strategies
--list-strategies			List strategies

Output includes throughput (RPS), latency percentiles (p50-p99), success/failure rates, and an ASCII histogram.

zc bench mesh emits a JSON mesh-warmup report with per-worker latency_p95, bandwidth_bps, and recommended_backpressure, suitable for jq or for feeding into operator tooling.

Remote Monitoring

Attach to a running broker from another machine:

zc attach localhost:9000
zc attach http://broker.mesh:9000

Renders the same TUI dashboard using the broker's /stats endpoint. Press r to force refresh.

System Diagnostics

zc info

Shows network info (hostname, mode, IPs), compute cluster availability (Ray, Dask, Spark), environment variables, and broker port status.

API Endpoints

The broker listens on 0.0.0.0:9000 by default.

Method	Endpoint	Auth	Description
GET	/health	None	Health check
GET	/stats	Bearer	Broker statistics (non-admin: own txns only)
GET	/workers	None	List registered workers
POST	/workers	X-Worker-Key	Register a worker
POST	/workers/heartbeat	X-Worker-Key	Worker heartbeat
DELETE	/workers/:id	X-Worker-Key	Unregister worker
POST	/execute	Bearer	Execute compute request
POST	/price	None	Estimate request price
GET	/credits/:user	Bearer	Get balance (own user or admin)
POST	/credits/:user/add	X-Api-Key (master)	Add credits
GET	/me	Bearer	Current user info
GET	/ledger/status	Bearer (admin)	Ledger status (API URL, connection state)
P2P Endpoints (ZAKURO_P2P=true)
POST	/peer/reserve	X-Peer-Key	Reserve credits for remote user
POST	/peer/commit	X-Peer-Key	Commit transaction after execution
POST	/peer/cancel	X-Peer-Key	Cancel reservation (refund)
POST	/peer/earn	X-Peer-Key	Credit earnings to broker owner
GET	/peer/balance	X-Peer-Key	Get balance for remote user
GET	/peer/workers	X-Peer-Key	List this broker's local workers
GET	/peer/health	X-Peer-Key	Peer broker health check

Execute Request Options

The /execute endpoint accepts a JSON body (or X-Zakuro-Requirements header) with:

Field	Type	Default	Description
cpus	float	1.0	Required vCPUs
memory_bytes	int	—	Required memory in bytes
gpus	int	0	Required GPUs
estimated_duration_secs	float	1.0	Estimated job duration (used for cost estimation)
budget_credits	float	—	Maximum credits to spend; request fails if cost would exceed this
strategy	string	best_price	Routing strategy
remote_only	bool	false	When true, skip local workers and dispatch only to peer brokers (requires ZAKURO_P2P=true)
worker_type	string	—	Filter workers by type (e.g. standard, premium)

Response headers include X-Zakuro-Worker (worker ID), X-Zakuro-Cost (credits spent), and X-Zakuro-Credits-Remaining (balance after execution).

Worker Registration

{
  "name": "worker-1",
  "uri": "http://10.13.13.5:8089",
  "worker_type": "zakuro",
  "resources": {
    "cpus_total": 8.0,
    "cpus_available": 8.0,
    "memory_total": 17179869184,
    "memory_available": 17179869184,
    "gpus_total": 1,
    "gpus_available": 1
  },
  "pricing": {
    "cpu_price": 0.001,
    "memory_price": 0.0001,
    "gpu_price": 0.01,
    "min_charge": 0.001
  }
}

Cluster Commands

Command	Description
zc up	Start 1 worker + broker locally (foreground)
zc up --workers N	Start N workers + broker
zc down	Stop all workers + broker started by zc up
zc workers	List workers registered with the local broker
zc update	Update CLI from get.zakuro-ai.com
zc pull	Pull updated Docker images
zc images	List Zakuro images
zc ps	List running containers
zc kill	Stop running containers
zc restart	Restart containers

Environment Variables

Variable	Required	Default	Description
ZAKURO_API_KEY	No*	—	Authentication token for dashboard API. *Optional when using only local or discovered brokers (P2P only).
ZAKURO_MASTER_KEY	No	—	Master API key for admin operations and standalone billing. When set (without ZAKURO_API_URL), the broker runs a fully local credit ledger with billing enforced.
ZAKURO_API_URL	No*	—	Dashboard API URL (e.g., https://my.zakuro-ai.com). *Required for dashboard-backed billing and worker sync. Omit to use standalone mode (ZAKURO_MASTER_KEY).
ZAKURO_WORKER_KEY	No	—	Shared secret for worker management endpoints (POST/DELETE /workers, heartbeat)
ZAKURO_WIREGUARD_IP	No	—	Override mesh IP detection (Docker sidecar setups). Named for the mesh we used to run; it applies to WireGuard too.
ZAKURO_PEERS	No	—	Comma-separated ip:port peers for explicit discovery (workers) or http://host:port (brokers in P2P mode)
ZAKURO_DISCOVER_BROKER_PEERS	No	true	When P2P is enabled and ZAKURO_PEERS is empty, scan localhost ports 9000–9010 for other brokers. Set to false to disable. When using discovered brokers, ZAKURO_API_KEY is optional.
ZAKURO_P2P	No	false	Enable P2P broker-to-broker communication
ZAKURO_PEER_KEY	No	—	Shared secret for P2P broker authentication (X-Peer-Key header)
ZAKURO_OWNER_ID	—	—	Not configurable. Always derived from ZAKURO_API_KEY (zk_{user_id}_{hex} format). Manual overrides are ignored.
ZAKURO_NODE_NAME	No	Auto-detected	Human-readable node name (falls back to the mesh hostname, then system hostname)
ZAKURO_WORKER_PORT	No	3960	Primary port workers listen on (discovery)
ZAKURO_SCAN_RANGE	No	—	Port range for discovery scan, e.g. 3960-3999
ZAKURO_SCAN_INTERVAL	No	15	Seconds between discovery scans

Development

Task Commands

This project uses Task as its task runner. Install it with go install github.com/go-task/task/v3/cmd/task@latest or see installation docs.

Command	Description
task build	Build binary and start Docker services
task build:bin	Compile release binary and install to /usr/local/bin/zc
task build:docker	Build Docker image only
task run	Build and run the CLI
task debug	Build and run with RUST_BACKTRACE=full
task release	Run the installed release binary
task broker	Start broker in foreground
task broker:tui	Start broker with TUI dashboard
task broker:daemon	Start broker in daemon mode
task bench	Run default benchmark
task bench:compare	Compare all routing strategies
task info	Show system diagnostics
task docker:up	Start Docker services
task docker:down	Stop Docker services
task docker:logs	Tail Docker service logs
task clean	Remove build artifacts

Docker

# Build and run with Docker Compose
task build

# Or build the image directly
docker build -t zakuro-ai/broker -f docker/Dockerfile .
docker run -p 9000:9000 -e ZAKURO_API_KEY=demo zakuro-ai/broker

Project Structure

src/
  main.rs              # CLI entry point and command routing
  broker/
    mod.rs             # Broker state, config, and per-worker local detection
    server.rs          # HTTP server (tiny_http) with execute handler
    worker.rs          # Worker registry, resources, and health tracking
    router.rs          # Routing strategies and worker selection
    credits.rs         # Lightweight per-user balance cache and rate-limit tracker (reservation/history owned by Ledger)
    ledger.rs          # Central ledger: API mode (dashboard) or local in-memory reserve-commit
    peer.rs            # P2P broker-to-broker communication (PeerClient, PeerManager, FNV-1a authority)
    flush.rs           # TransactionBuffer for batched writes (5s cycle, via dashboard API)
    discovery.rs       # mesh/local/peer-based worker discovery
    wal.rs             # Write-ahead log (JSONL + DashMap index, batched fsync)
    recovery.rs        # WAL replay on startup (reserved→cancel, executed→commit)
    tui.rs             # Interactive terminal dashboard (ratatui)
    bench.rs           # Benchmarking tool
    stats.rs           # Statistics tracking
    info.rs            # System diagnostics
  common/mod.rs        # Shared utilities
  exec/mod.rs          # Command execution helpers
  manager/mod.rs       # Docker container management
  network/mod.rs       # Network interface detection
  envs/mod.rs          # Environment variable handling

Troubleshooting

• "You are missing ZAKURO_API_KEY" — set ZAKURO_API_KEY for dashboard auth. For standalone billing, set ZAKURO_MASTER_KEY instead. P2P-only setups (auto-discovered peers) don't require either.
• No workers discovered — ensure workers expose /health and /info endpoints on port 3960 (or ZAKURO_WORKER_PORT). Check with zc info.
• Credit operations failing — ensure both ZAKURO_API_KEY and ZAKURO_API_URL are set for dashboard mode. For standalone mode, set only ZAKURO_MASTER_KEY. Verify the dashboard returns the credits_balance field in /api/broker/balance/{id}.
• Zero cost on remote executions — ensure is_billing_enabled() returns true: either dashboard URL+key or ZAKURO_MASTER_KEY must be set. Local workers (127.0.0.1/localhost) are always free; remote mesh workers are billed.
• remote_only: true returns 503 — ensure ZAKURO_P2P=true and at least one peer broker is reachable. Without peers, remote_only requests cannot be served.
• P2P mode not working — ensure ZAKURO_P2P=true, ZAKURO_PEER_KEY matches on all brokers. Use ZAKURO_PEERS for explicit URLs, or leave empty to auto-discover brokers on localhost (ports 9000–9010).
• Mesh IP not detected — the broker recognises a zakuro0 or wg* interface holding an address in 10.13.13.0/24 (see get_mesh_ip in src/broker/discovery.rs). An interface outside that subnet is ignored. Set ZAKURO_WIREGUARD_IP to override.
• TUI not rendering — ensure your terminal supports 256 colors and is at least 80 columns wide.

Documentation

See docs/USAGE.md for the complete usage guide covering all API details, configuration reference, and the credit system in depth.

See docs/QUIC_TRANSPORT.md for details on the QUIC-based P2P transport layer.

License

MIT