# Forge Orchestration
**Rust-Native Orchestration Platform for Distributed Workloads**
[](https://crates.io/crates/forge-orchestration)
[](https://docs.rs/forge-orchestration)
[](LICENSE)
[](https://www.rust-lang.org)
A high-performance orchestration platform for Rust, designed to manage distributed workloads at hyper-scale — with intelligent bin-packing, a closed reconcile → schedule → bind → persist control loop, and simulation/agent-native scheduling primitives (gang co-scheduling and tick-deadline ordering).
## Performance Benchmarks
Measured on an **Intel Core i9-10980XE (18C/36T @ 3.0 GHz), Windows 11, `rustc 1.95.0`**,
release build via Criterion (30 samples, 2 s measurement). These are **absolute
throughput numbers on one machine — not a comparison against Kubernetes** (Forge does
not benchmark K8s here), and as wall-clock microbenchmarks on a non-isolated machine they
vary run-to-run (≈1.5×). Throughput = workloads ÷ median per-batch time. Reproduce:
```bash
cargo bench --bench bind_cycle_benchmark --bench scheduler_benchmark
```
### Honest bind cycle — `OptimizedScheduler`, sequential commit path
Each call finds the best node **and commits** the allocation (decrements capacity in
the node *and* the score cache). This is the number to quote for "how fast can we
actually schedule **and bind** a workload?"
| 10 nodes | 0.22 µs (~4.6M/s) | 1.12 µs (~890K/s) | — |
| 50 nodes | 0.53 µs (~1.9M/s) | 1.73 µs (~580K/s) | 0.60 µs (~1.7M/s) |
| 100 nodes | 1.05 µs (~950K/s) | 1.95 µs (~510K/s) | 1.10 µs (~910K/s) |
| 500 nodes | 4.78 µs (~210K/s) | — | 4.77 µs (~210K/s) |
¹ adds `serde_json` serialization of the bind record + a `MemoryStore` write per
workload (approximates the etcd/state-store write a real control plane pays). The
**~510K binds/s at 100 nodes** is the most realistic single-workload figure.
² `schedule_and_commit_batch` over 500 workloads.
### Monolithic `Scheduler` — full affinity / taint / preemption semantics
Fresh cluster, 500 workloads, per-workload median:
| bin-pack | 1.97 µs | ~507K/s |
| spread | 1.96 µs | ~510K/s |
| GPU-locality (20 nodes × 8 A100, GPU workloads) | 3.12 µs | ~320K/s |
Online `LearnedScheduler` score + feedback update: **~0.15 µs (~6.8M updates/s)**.
### Resolved: `schedule_fast` is now sequential (an earlier Rayon path was a pessimization)
`OptimizedScheduler::schedule_fast` previously switched to Rayon `par_iter` above 16
nodes. Benchmarks showed that was a **40–300× pessimization** for the trivial integer
scoring — the per-call fan-out cost dwarfed the work. The Rayon path (and the `rayon`
dependency) has been **removed**; `schedule_fast` now uses the same sequential scan as
`schedule_fast_commit`. Measured `score_only`, before vs after:
| 50 nodes | 28.9 µs/wl (~35K/s) | 0.57 µs/wl (~1.8M/s) |
| 100 nodes | 36.9 µs/wl (~27K/s) | 1.05 µs/wl (~950K/s) |
| 500 nodes | 56.1 µs/wl (~18K/s) | 4.88 µs/wl (~205K/s) |
It now tracks `schedule_commit` within noise (scoring without committing does marginally
less work). This also retired the earlier "10–2000× faster than K8s / 1M decisions/sec"
numbers, which came from the non-committing scoring loop and did not reflect a real bind.
### Genuine performance properties (confirmed by the numbers above)
- **Integer-only scoring** — the hot path is allocation-free `u32` math, no floating point.
- **Pre-computed score cache** — `schedule_fast_commit` reads/decrements a cached view in O(nodes).
- **Sequential committing bind** scales smoothly: ~1 µs/bind at 100 nodes, ~5 µs at 500 (no Rayon).
- **First-Fit Decreasing bin-packing** for utilization.
## Features
| **High-Performance Scheduler** | Parallel bin-packing / spread / GPU-locality scoring; allocations commit to node state |
| **Reconcile Control Loop** | Converges desired jobs onto actual assignments: schedule, persist-as-truth, release on scale-down / node loss, execute autoscaling |
| **Sim/Agent-Native Scheduling** | `SimCell` (world shard + agent policies as one unit), all-or-nothing **gang** co-scheduling, **tick-deadline** (EEVDF-style) ordering |
| **Control Plane** | Kubernetes-style API server with admission controllers and **SSE** watch streams |
| **Durable State** | In-memory / file (default); **multi-node Raft** via openraft over an HTTP transport (opt-in `raft`) with replication + leader failover, and a **crash-durable** disk-backed log + snapshots via [fjall](https://github.com/fjall-rs/fjall) (opt-in `raft-persist`) |
| **Multi-Region Federation** | Geo-aware routing and latency-based failover (cross-region replication is experimental / in-memory) |
| **MoE Routing** | Intelligent request routing with load-aware, GPU-aware, and version-aware strategies |
| **Autoscaling** | Threshold-based and target-utilization policies with hysteresis, executed by the reconcile loop |
| **Resilience** | Circuit breakers and exponential-backoff retry (opt-in on the Nomad client) |
| **Game Server SDK** | UDP/TCP port allocation, session management, spot instance handling |
| **AI/ML Inference** | Request batching, SSE streaming for LLM tokens |
## Use Cases
Forge is built for the workload mainstream orchestrators were never designed for —
**AI models acting inside simulations at scale** — and is competent at the ordinary
distributed-workload jobs around it. The *Status* notes are deliberate: nothing below
is claimed beyond what the test suite actually exercises.
### 🌍 Agents living in simulations — the reason Forge exists
Co-schedule a **world shard and its agents' policy-inference models as one unit**
([`SimCell`](forge/src/scheduler/sim.rs)): **gang-place** them on the same node /
interconnect-local GPUs so the world↔agent tensor exchange never pays the cross-node
tax, and order them by **tick deadline** so the world and its agents hit the same
simulation frame — neither stalls the other. Spatial **interest management**
(`Region3D`) decides which cells even need GPU locality. This is the primitive
Borg/Kubernetes have no concept of.
*Status: primitives implemented and tested in-process; real multi-node GPU validation
is the next milestone.*
### 🧬 Gang-scheduled distributed training / inference
All-or-nothing placement — N workers land together or **none** do — eliminating the
idle-GPU waste where some ranks get slots while the rest queue, burning accelerators
that produce nothing.
### ⏱️ Deadline-driven, real-time tick workloads
EEVDF-style **virtual-deadline** scheduling where the deadline *is* the next frame/tick
— for fixed-cadence simulations, RL rollouts, or latency-bound batch steps. Missed
ticks are first-class (`Late` / `Drop` / `Backpressure`).
### 🔁 A self-driving service & batch control plane
Submit jobs; the **reconcile loop** converges desired → actual: it schedules, persists
the binding as the source of truth, restarts/reschedules on failure or node loss, and
**executes** autoscaling decisions (not just computes them) — a Borg-style control loop
in a single Rust binary with no GC pauses in the hot path.
### 🗄️ Durable, fault-tolerant control-plane state
Run the state store as a **multi-node Raft cluster over HTTP** (`raft`) that replicates
writes and elects a new leader when one fails, backed by a **crash-durable** disk log +
snapshots (`raft-persist`) that survive a process restart.
*Status: replication, leader failover, and restart-survival are tested in-process
(3 nodes on localhost); not yet hardened at real cluster scale.*
### 🧠 Inference-fleet request routing
Route requests to experts by **load / GPU availability / model version** (MoE routers),
with dynamic request **batching** and **SSE** token streaming for LLM serving.
### 🎮 Game-server & spot fleets
UDP/TCP port allocation, session lifecycle, and spot-instance handling via the workload
SDK.
## Installation
```toml
[dependencies]
forge-orchestration = "0.6"
tokio = { version = "1", features = ["full"] }
```
## Quick Start
### Control Plane
```rust
use forge_orchestration::{ForgeBuilder, AutoscalerConfig, Job, Task, Driver};
#[tokio::main]
async fn main() -> forge_orchestration::Result<()> {
// Build the orchestrator
let forge = ForgeBuilder::new()
.with_autoscaler(AutoscalerConfig::default()
.upscale_threshold(0.8)
.downscale_threshold(0.3))
.build()?;
// Define and submit a job
let job = Job::new("my-service")
.with_group("api", Task::new("server")
.driver(Driver::Exec)
.command("/usr/bin/server")
.args(vec!["--port", "8080"])
.resources(500, 256));
forge.submit_job(job).await?;
// Run the control plane
forge.run().await?;
Ok(())
}
```
### Workload SDK
The SDK is included in the main crate under `forge_orchestration::sdk`:
```rust
use forge_orchestration::sdk::{ready, allocate_port, graceful_shutdown, shutdown_signal};
#[tokio::main]
async fn main() -> forge_orchestration::Result<()> {
// Signal readiness to orchestrator
ready()?;
// Allocate a port dynamically
let port = allocate_port(8000..9000)?;
println!("Listening on port {}", port);
// Install graceful shutdown handlers
graceful_shutdown();
// ... your server logic ...
// Wait for shutdown signal
shutdown_signal().await;
Ok(())
}
```
## Architecture
```
[User App] --> [Forge SDK] (ready(), allocate(), shutdown())
|
v
[Forge Control Plane]
- Tokio Runtime (async loops + reconcile control loop)
- Scheduler (bin-pack / spread / GPU-locality; gang + tick-deadline for sim cells)
- Raft (multi-node consensus over HTTP, opt-in `raft` feature)
- State: in-memory / file (default); multi-node Raft (opt-in `raft`), with a
crash-durable fjall-backed log + snapshot (opt-in `raft-persist`)
- MoE Router (gating to experts)
|
v
[Nomad Scheduler] (jobs: containers/binaries)
|
v
[Workers/Nodes]
- QUIC/TLS Networking
- Prometheus Metrics
```
## API Reference
### Modules
| Module | Description |
|--------|-------------|
| `job` | `Job`, `Task`, `TaskGroup`, `Driver` definitions |
| `moe` | `MoERouter` trait, `DefaultMoERouter`, `LoadAwareMoERouter`, `RoundRobinMoERouter` |
| `autoscaler` | `Autoscaler`, `AutoscalerConfig`, `ScalingPolicy` trait |
| `nomad` | `NomadClient` for HashiCorp Nomad API |
| `storage` | `StateStore` trait, `MemoryStore`, `FileStore` |
| `networking` | `HttpServer`, `QuicTransport` |
| `metrics` | `ForgeMetrics`, `MetricsExporter`, `MetricsHook` trait |
| `sdk` | Workload SDK: `ready()`, `allocate_port()`, `graceful_shutdown()`, `ForgeClient` |
### MoE Routing
Built-in routers:
- **`DefaultMoERouter`**: Hash-based consistent routing
- **`LoadAwareMoERouter`**: Routes to least-loaded expert with affinity
- **`RoundRobinMoERouter`**: Sequential distribution
Custom router:
```rust
use forge_orchestration::moe::{MoERouter, RouteResult};
use async_trait::async_trait;
struct MyRouter;
#[async_trait]
impl MoERouter for MyRouter {
async fn route(&self, input: &str, num_experts: usize) -> RouteResult {
RouteResult::new(input.len() % num_experts)
}
fn name(&self) -> &str { "my-router" }
}
```
### Autoscaling
```rust
use forge_orchestration::AutoscalerConfig;
let config = AutoscalerConfig::default()
.upscale_threshold(0.8)
.downscale_threshold(0.3)
.hysteresis_secs(300)
.bounds(1, 100);
```
### Storage
```rust
use forge_orchestration::storage::{MemoryStore, FileStore};
let memory = MemoryStore::new();
let file = FileStore::open("/var/lib/forge/state.json")?;
```
### Metrics
```rust
use forge_orchestration::ForgeMetrics;
let metrics = ForgeMetrics::new()?;
metrics.record_job_submitted();
metrics.record_scale_event("my-job", "up");
let text = metrics.gather_text()?;
```
### SDK Functions
| `sdk::ready()` | Signal readiness to orchestrator |
| `sdk::allocate_port(range)` | Allocate an available port from range |
| `sdk::release_port(port)` | Release an allocated port |
| `sdk::graceful_shutdown()` | Install SIGTERM/SIGINT handlers |
| `sdk::shutdown_signal()` | Async wait for shutdown signal |
| `sdk::ForgeClient` | HTTP client for Forge API |
## Environment Variables
| `FORGE_API` | Forge API endpoint for SDK |
| `FORGE_ALLOC_ID` | Allocation ID (set by orchestrator) |
| `FORGE_TASK_NAME` | Task name (set by orchestrator) |
## Builder Configuration
```rust
use forge_orchestration::ForgeBuilder;
ForgeBuilder::new()
.with_nomad_api("http://localhost:4646")
.with_nomad_token("secret-token")
.with_store_path("/var/lib/forge/state.json")
.with_node_name("forge-1")
.with_datacenter("dc1")
.with_autoscaler(AutoscalerConfig::default())
.with_metrics(true)
.build()?
```
## License
Apache 2.0