Crate crabka_rebalancer 

Crabka rebalancer — Cruise-Control-equivalent partition placement advisor and executor.

The crate ingests broker/topic metrics, builds a cluster model, evaluates goal plugins, and emits executable partition-reassignment plans. The Connect-RPC API exposes the same analysis and execution path used by the Kubernetes operator.

Optimization workflow

use crabka_rebalancer::capacity::BrokerCapacities;
use crabka_rebalancer::goals::{GoalContext, leader_distribution::LeaderDistribution};
use crabka_rebalancer::model::{BrokerView, ClusterState, PartitionView};
use crabka_rebalancer::optimizer;
use crabka_rebalancer::scraper::UsageStore;
use std::sync::Arc;

let state = ClusterState {
    cluster_id: Some("cluster-a".into()),
    snapshot_at_ms: 1_713_000_000_000,
    brokers: vec![
        BrokerView {
            id: 1,
            host: "b1".into(),
            port: 9092,
            rack: None,
        },
        BrokerView {
            id: 2,
            host: "b2".into(),
            port: 9092,
            rack: None,
        },
    ],
    partitions: vec![PartitionView {
        topic: "orders".into(),
        partition: 0,
        replicas: vec![1, 2],
        leader: 1,
        isr: vec![1, 2],
    }],
    in_flight_reassignments: Vec::new(),
};
let ctx = GoalContext {
    imbalance_threshold_pct: 10,
    max_movements_per_proposal: 100,
    min_topic_leaders_per_broker: 0,
    broker_capacities: Arc::new(BrokerCapacities::default()),
    broker_usages: Arc::new(UsageStore::default()),
};
let goal = LeaderDistribution;
let out = optimizer::optimize(&state, &[&goal], &ctx)?;
println!("{} partition movements", out.proposal.movements.len());

Modules

api

Connect-RPC service wiring.

capacity

Per-broker capacity configuration. Loaded from a YAML file at startup; threaded into GoalContext so the capacity goals can enforce operator-supplied limits.

detector

Anomaly detector.

executor

Execute-path state machine. Executor runs one Execution at a time against the cluster via a ClientFacade.

goals

Goal trait and shared context. Concrete goals live in sibling modules.

health

Plain axum routes for /healthz, /readyz, /metrics. Mounted alongside the Connect-RPC router by the binary entry.

ingest

Periodic cluster-state snapshotter. Spawned by the binary entry; writes the latest snapshot into an ArcSwap<Option<ClusterState>> that the RPC handlers read.

metrics

Process-level OpenMetrics surface for the rebalancer.

model

In-memory data model for the rebalancer.

optimizer

Optimizer: runs an ordered list of Goals over a ClusterState, coalesces their movements, and emits a Proposal.

pb

Generated protobuf + Connect server stubs. The actual content lives in OUT_DIR/crabka.rebalancer.v1.rs and is produced by build.rs.

scraper

Per-partition metric scraper. Spawned from the binary entry when --metrics-scrape-targets is non-empty.

state_topic

Rebalancer state persistence via an internal compacted topic on the Crabka cluster being managed. Replaces the file-backed {data_dir}/in_flight.json store. Survives pod restart; prerequisite for multi-replica HA.