bmux_cluster_plugin

Bundled server clusters plugin for bmux.

Overview

This crate owns the bmux.cluster plugin domain.

Current scope:

• Durable federation identity foundation: activation creates or validates a local Ed25519 node key; NodeId is the canonical public-key identity, and cluster init creates an idempotent persistent ClusterId through the generated cluster-command/v1:init service. Corrupt, future-version, or mismatched identity records fail closed without key rotation.
• Generated membership lifecycle commands: cluster enrollment-token create, cluster join, cluster leave, and cluster members. Enrollment tokens are issuer-signed, expire within the reviewed ten-minute maximum, are request-idempotent, and are single-use across node identities; same-node redemption is retry-safe. The candidate signs token-, endpoint-, identity-, and protocol-bound possession claims before redemption. Join rejects incompatible wire epochs, peer revisions, durable schema ranges, and mandatory feature sets before membership mutation, then persists an issuer-signed, expiring public membership credential while private keys remain local. Join routes generated redemption through the explicit bmux.connections endpoint and validates the issuer/member snapshot before local adoption. Leave uses a signed prepare/accept/commit transaction and preserves the local node key.
• Generated peer authentication: cluster-peer-auth/v1 issues verifier-signed 30-second challenges bound to cluster, verifier credential, claimant audience, and nonce; the claimant verifies the verifier's active credential and signs the full challenge; the verifier checks the claimant's active credential and consumes the nonce durably. Join and remote leave orchestration require this challenge/prove/authenticate flow through bmux.connections, so the same plugin-owned authentication is transport-neutral. Replays, forged proofs, wrong audience, expiry, stale credentials, inactive members, and challenge tampering fail closed. Isolated two-server process tests prove the real join and leave path over SSH and quick-TLS.
• Durable node capability metadata separates the exclusive consensus role (voter or observer-edge) from independent worker and ingress grants. Initializers default to voter+worker+ingress; enrollment defaults to observer-edge+worker. Explicit grants are signed into enrollment tokens and validated before adoption. Legacy member records are migrated deterministically.
• Read-only cluster inventory and health checks (cluster hosts/status/doctor)
• Inventory sourced from [plugins.settings."bmux.cluster"].clusters
• Target resolution validated against [connections.targets]
• Readiness probes delegated to core remote commands (remote test / remote doctor)
• cluster up creates/reuses cluster-<name> session and launches host-bound panes
• Partial-start semantics: unhealthy hosts are reported as degraded while healthy hosts still launch
• cluster up supports launch failure policy controls (--retries, --on-failure=continue|abort|prompt)
• cluster events shows connection lifecycle events (--format text|json, --cluster, --target, --state, --since, --limit); --since accepts now/0, unix ms, or relative durations like 15m, 2h, 1h30m
• cluster pane new creates an ad-hoc host-bound pane via the generic pane launch API
• cluster pane move relocates a pane to a destination host and retargets pane naming
• cluster pane retry relaunches a host-bound pane by inferring target from pane naming convention
• Cluster pane target metadata is persisted in plugin storage for robust move/retry behavior
• Connection lifecycle state (connecting/retrying/degraded/failed) is tracked in pane metadata
• Panes transition to ready only after post-launch health verification succeeds
• cluster pane retry supports probe retry policy controls (--retries, --on-failure=abort|continue|prompt)
• cluster pane new|move|retry accept --cluster for deterministic gateway routing; in multi-cluster layouts, commands hard-fail when cluster inference is ambiguous
• Gateway overrides are available on cluster commands: --gateway, --gateway-mode=auto|direct|pinned, --gateway-policy=balanced|aggressive|conservative, and --gateway-no-failover
• Routed cluster commands support --dry-run to run real gateway probes/selection without executing the cluster operation itself (--format text|json)
• Routed/special gateway commands support --why for concise decision summaries in text and JSON (decision_summary)
• cluster gateway status reports effective gateway mode, candidate order, preferred gateway cache, cooldown state, and selected candidate hint (--format text|json)
• cluster gateway explain probes candidates and explains why selection would succeed/fail without mutating gateway runtime cache/cooldown state (--format text|json)
• cluster gateway doctor inspects gateway candidate health and emits actionable findings (--format text|json)
• cluster gateway why provides one-shot routing diagnosis, top actions, and recent history (--format text|json)
• cluster gateway history shows recent gateway routing decisions and observations (--format text|json, --since <duration>, --limit <count>, --result, --reason, --candidate, --command)
• cluster gateway history-export emits machine-friendly history output (--format json|ndjson)
• cluster gateway history-clear clears history for one cluster or all clusters (--cluster/--all, filter flags, --confirm for broad non-interactive clears)
• cluster gateway reset clears persisted gateway runtime state for one cluster (--cluster) or all clusters (--all)
• In multi-cluster setups, cluster gateway status|explain require explicit cluster selection (--cluster or positional cluster name)
• Cluster service interfaces are implemented for query/command/event-list integrations

Failure Semantics

• cluster up --on-failure=abort stops launching additional hosts after the first terminal failure but preserves any panes already launched successfully.
• cluster up --on-failure=continue keeps launching remaining hosts and reports failed hosts as degraded.
• cluster up --on-failure=prompt asks for interactive retry/continue/abort decisions when prompt runtime is available; if unavailable, it safely falls back to abort.

Gateway Failover Mode

• Cluster commands can execute through a cluster member gateway before orchestration runs. This is intended for topologies where private cluster members are reachable from peers but not directly from the caller machine.
• gateway_mode defaults to auto for each cluster:
  • auto: try gateway candidates in order until one succeeds.
  • direct: run locally (no gateway indirection).
  • pinned: require a single configured gateway_target.
• In auto, candidates come from gateway_candidates when provided; otherwise they fall back to cluster targets.
• If every gateway candidate fails, cluster command execution hard-fails (no silent direct fallback).
• Auto mode tracks persisted last-known-good/cooldown state, per-candidate stability stats, and breaker state.
• Breaker behavior defaults: open after 3 consecutive failures, then half-open retry after cooldown.
• Cooldown is adaptive per candidate (failure streak increases delay up to cooldown_max_ms) and resets on successful execution.
• Cooldown handling is reason-class-aware (auth/service denial back off faster than transient network failures) with bounded jitter (cooldown_jitter_pct) to reduce herd retries.
• Half-open probation requires consecutive successes (breaker_half_open_required_successes) before closing breaker again.
• History retention and size are policy-driven (history_retention_ms, history_max_entries).
• Policy presets tune breaker/cooldown/probe defaults:
  • balanced: current defaults.
  • aggressive: faster failover and shorter cooldown.
  • conservative: slower failover and longer stabilization tabs.
• Candidate ordering in auto mode is stability-first (latency used as tie-break), with explicit skip reasons (cooldown / breaker_open) visible in status/explain and dry-run output.
• Text output may truncate long candidate names for alignment only; JSON keeps full candidate values.

Gateway Output Examples

Text table columns are aligned and consistent across cluster gateway status, cluster gateway explain, and routed command --dry-run:

candidate                 preferred stability  breaker    cooldown_ms  ok    reason         latency_ms skip           detail
prod-a                    true      5000       closed     -            true  ok             42         -              gateway command bridge reachable
prod-b                    false     200000     open       29000        false connect        15         breaker_open   connection refused

JSON example for cluster gateway explain --format json:

{
  "cluster": "prod",
  "command": "cluster-gateway-explain",
  "mode": "auto",
  "no_failover": false,
  "selected_candidate": "prod-a",
  "result": "success",
  "would_mutate": {
    "enabled": false,
    "last_good": false,
    "cooldown": false,
    "breaker": false,
    "persistence_write": false
  },
  "probes": [
    {
      "candidate": "prod-a",
      "preferred": true,
      "stability_score": 5000,
      "breaker_state": "closed",
      "cooldown_ms": null,
      "skip_reason": null,
      "historical_latency_ms": 42,
      "ok": true,
      "reason": "ok",
      "latency_ms": 42,
      "detail": "gateway command bridge reachable"
    }
  ],
  "failures": []
}

JSON example for routed command dry-run (cluster status prod --dry-run --format json):

{
  "cluster": "prod",
  "command": "cluster-status",
  "mode": "auto",
  "result": "failure",
  "selected_candidate": null,
  "would_mutate": {
    "enabled": false,
    "last_good": false,
    "cooldown": false,
    "breaker": false,
    "persistence_write": false
  },
  "probes": [],
  "failures": []
}

Example:

[plugins.settings."bmux.cluster"]
# Stable connections target advertised for authenticated consensus RPC.
# This may be a self-contained tls://, https://, iroh://, or ssh:// URI, or a
# named [connections.targets] alias provisioned identically on every cluster
# member. Local targets are forbidden. Aliases are part of cluster provisioning:
# the same alias must resolve to the same remote identity on every node.
consensus_endpoint = "prod-a"

[plugins.settings."bmux.cluster".clusters.prod]
targets = ["prod-a", "prod-b", "prod-c"]
gateway_mode = "auto"
# optional; defaults to targets when omitted
# gateway_candidates = ["prod-a", "prod-b"]

Commands

• cluster init --endpoint <advertised-target>
• cluster enrollment-token create --endpoint <target> [--ttl-ms <milliseconds>] [--role voter|observer-edge] [--worker|--no-worker] [--ingress|--no-ingress]
• cluster join <token> [--issuer <target>] [--endpoint <advertised-target>]
• cluster leave
• cluster members
• cluster up
• cluster status
• cluster doctor
• cluster hosts
• cluster events
• cluster pane new
• cluster pane move
• cluster pane retry
• cluster gateway status
• cluster gateway explain
• cluster gateway doctor
• cluster gateway why
• cluster gateway history
• cluster gateway history-export
• cluster gateway history-clear
• cluster gateway reset

Services

• cluster-query/v1
• cluster-command/v1
• cluster-worker-command/v1 — generated launch/adopt/input/resize/signal/restart/close contracts; every mutation carries typed execution identity, generation, control term, and bounded lease authority.
• cluster-worker-state/v1 — generated execution query/inventory plus cursor-based output and full terminal snapshot repair contracts.
• cluster-peer-auth/v1
• cluster-connection-events/v1

Service Contract Notes

• cluster-query/v1
  • identity returns the local public node identity, optional current cluster ID, and explicit wire/peer/schema/plugin/feature compatibility offer.
  • members returns durable public membership, signed credential validity/issuer metadata, negotiated compatibility, and member states.
  • list_clusters returns settings-resolved cluster inventory.
  • status returns host states from probe execution (ready or degraded).
  • Errors use list_clusters_failed / status_failed.
• cluster-command/v1
  • init creates or returns the durable cluster identity; --endpoint advertises the stable connections target required by voter consensus startup. Advertised endpoints are either self-contained tls://, https://, iroh://, or ssh:// references, or named [connections.targets] aliases provisioned identically across the cluster. Blank/local/malformed references, duplicate active-member endpoints, and implicit active-voter endpoint rewrites are rejected. Every authenticated connection verifies that the resolved endpoint presents the expected member NodeId. The same endpoint may be configured as plugins.settings."bmux.cluster".consensus_endpoint for restart-safe activation.
  • enrollment_token_create, redeem_enrollment, and join implement signed, expiring, retry-safe enrollment as generated typed phases; redemption verifies node-key possession and compatibility before consuming the token, then returns an issuer-signed public membership credential.
  • leave_prepare, accept_leave, and leave implement signed, retryable leave coordination without nested service dispatch.
  • up returns session id plus per-host launch status payload.
  • pane_new, pane_retry, and pane_move return operation result payloads with pane/session ids.
  • Errors use up_failed, pane_new_failed, pane_retry_failed, and pane_move_failed.
• cluster-peer-auth/v1
  • challenge, prove, and authenticate provide mutually verified active-member authentication with audience, expiry, credential-serial, signature, and durable single-use nonce checks.
• cluster-connection-events/v1
  • list returns persisted lifecycle events ring buffer for cluster connections.
  • Errors use connection_events_list_failed.