Forge Orchestration

Rust-Native Orchestration Platform for Distributed Workloads

A high-performance orchestration platform for Rust, designed to manage distributed workloads at hyper-scale — with intelligent bin-packing, a closed reconcile → schedule → bind → persist control loop, and simulation/agent-native scheduling primitives (gang co-scheduling and tick-deadline ordering).

Performance Benchmarks

Measured on an Intel Core i9-10980XE (18C/36T @ 3.0 GHz), Windows 11, rustc 1.95.0, release build via Criterion (30 samples, 2 s measurement). These are absolute throughput numbers on one machine — not a comparison against Kubernetes (Forge does not benchmark K8s here), and as wall-clock microbenchmarks on a non-isolated machine they vary run-to-run (≈1.5×). Throughput = workloads ÷ median per-batch time. Reproduce:

cargo bench --bench bind_cycle_benchmark --bench scheduler_benchmark

Honest bind cycle — OptimizedScheduler, sequential commit path

Each call finds the best node and commits the allocation (decrements capacity in the node and the score cache). This is the number to quote for "how fast can we actually schedule and bind a workload?"

¹ adds serde_json serialization of the bind record + a MemoryStore write per workload (approximates the etcd/state-store write a real control plane pays). The ~510K binds/s at 100 nodes is the most realistic single-workload figure. ² schedule_and_commit_batch over 500 workloads.

Monolithic Scheduler — full affinity / taint / preemption semantics

Fresh cluster, 500 workloads, per-workload median:

Online LearnedScheduler score + feedback update: ~0.15 µs (~6.8M updates/s).

Resolved: schedule_fast is now sequential (an earlier Rayon path was a pessimization)

OptimizedScheduler::schedule_fast previously switched to Rayon par_iter above 16 nodes. Benchmarks showed that was a 40–300× pessimization for the trivial integer scoring — the per-call fan-out cost dwarfed the work. The Rayon path (and the rayon dependency) has been removed; schedule_fast now uses the same sequential scan as schedule_fast_commit. Measured score_only, before vs after:

It now tracks schedule_commit within noise (scoring without committing does marginally less work). This also retired the earlier "10–2000× faster than K8s / 1M decisions/sec" numbers, which came from the non-committing scoring loop and did not reflect a real bind.

Genuine performance properties (confirmed by the numbers above)

• Integer-only scoring — the hot path is allocation-free u32 math, no floating point.
• Pre-computed score cache — schedule_fast_commit reads/decrements a cached view in O(nodes).
• Sequential committing bind scales smoothly: ~1 µs/bind at 100 nodes, ~5 µs at 500 (no Rayon).
• First-Fit Decreasing bin-packing for utilization.

Features

Use Cases

Forge is built for the workload mainstream orchestrators were never designed for — AI models acting inside simulations at scale — and is competent at the ordinary distributed-workload jobs around it. The Status notes are deliberate: nothing below is claimed beyond what the test suite actually exercises.

🌍 Agents living in simulations — the reason Forge exists

Co-schedule a world shard and its agents' policy-inference models as one unit (SimCell): gang-place them on the same node / interconnect-local GPUs so the world↔agent tensor exchange never pays the cross-node tax, and order them by tick deadline so the world and its agents hit the same simulation frame — neither stalls the other. Spatial interest management (Region3D) decides which cells even need GPU locality. This is the primitive Borg/Kubernetes have no concept of. Status: primitives implemented and tested in-process; real multi-node GPU validation is the next milestone.

🧬 Gang-scheduled distributed training / inference

All-or-nothing placement — N workers land together or none do — eliminating the idle-GPU waste where some ranks get slots while the rest queue, burning accelerators that produce nothing.

⏱️ Deadline-driven, real-time tick workloads

EEVDF-style virtual-deadline scheduling where the deadline is the next frame/tick — for fixed-cadence simulations, RL rollouts, or latency-bound batch steps. Missed ticks are first-class (Late / Drop / Backpressure).

🔁 A self-driving service & batch control plane

Submit jobs; the reconcile loop converges desired → actual: it schedules, persists the binding as the source of truth, restarts/reschedules on failure or node loss, and executes autoscaling decisions (not just computes them) — a Borg-style control loop in a single Rust binary with no GC pauses in the hot path.

🗄️ Durable, fault-tolerant control-plane state

Run the state store as a multi-node Raft cluster over HTTP (raft) that replicates writes and elects a new leader when one fails, backed by a crash-durable disk log + snapshots (raft-persist) that survive a process restart. Status: replication, leader failover, and restart-survival are tested in-process (3 nodes on localhost); not yet hardened at real cluster scale.

🧠 Inference-fleet request routing

Route requests to experts by load / GPU availability / model version (MoE routers), with dynamic request batching and SSE token streaming for LLM serving.

🎮 Game-server & spot fleets

UDP/TCP port allocation, session lifecycle, and spot-instance handling via the workload SDK.

Installation

[dependencies]

forge-orchestration = "0.6"

tokio = { version = "1", features = ["full"] }

Quick Start

Control Plane

use forge_orchestration::{ForgeBuilder, AutoscalerConfig, Job, Task, Driver};

#[tokio::main]
async fn main() -> forge_orchestration::Result<()> {
    // Build the orchestrator
    let forge = ForgeBuilder::new()
        .with_autoscaler(AutoscalerConfig::default()
            .upscale_threshold(0.8)
            .downscale_threshold(0.3))
        .build()?;

    // Define and submit a job
    let job = Job::new("my-service")
        .with_group("api", Task::new("server")
            .driver(Driver::Exec)
            .command("/usr/bin/server")
            .args(vec!["--port", "8080"])
            .resources(500, 256));

    forge.submit_job(job).await?;

    // Run the control plane
    forge.run().await?;
    Ok(())
}

Workload SDK

The SDK is included in the main crate under forge_orchestration::sdk:

use forge_orchestration::sdk::{ready, allocate_port, graceful_shutdown, shutdown_signal};

#[tokio::main]
async fn main() -> forge_orchestration::Result<()> {
    // Signal readiness to orchestrator
    ready()?;

    // Allocate a port dynamically
    let port = allocate_port(8000..9000)?;
    println!("Listening on port {}", port);

    // Install graceful shutdown handlers
    graceful_shutdown();

    // ... your server logic ...

    // Wait for shutdown signal
    shutdown_signal().await;
    Ok(())
}

Architecture

[User App] --> [Forge SDK] (ready(), allocate(), shutdown())
              |
              v
[Forge Control Plane]
  - Tokio Runtime (async loops + reconcile control loop)
  - Scheduler (bin-pack / spread / GPU-locality; gang + tick-deadline for sim cells)
  - Raft (multi-node consensus over HTTP, opt-in `raft` feature)
  - State: in-memory / file (default); multi-node Raft (opt-in `raft`), with a
    crash-durable fjall-backed log + snapshot (opt-in `raft-persist`)
  - MoE Router (gating to experts)
  |
  v
[Nomad Scheduler] (jobs: containers/binaries)
  |
  v
[Workers/Nodes]
  - QUIC/TLS Networking
  - Prometheus Metrics

API Reference

Modules

MoE Routing

Built-in routers:

• DefaultMoERouter: Hash-based consistent routing
• LoadAwareMoERouter: Routes to least-loaded expert with affinity
• RoundRobinMoERouter: Sequential distribution

Custom router:

use forge_orchestration::moe::{MoERouter, RouteResult};
use async_trait::async_trait;

struct MyRouter;

#[async_trait]
impl MoERouter for MyRouter {
    async fn route(&self, input: &str, num_experts: usize) -> RouteResult {
        RouteResult::new(input.len() % num_experts)
    }
    fn name(&self) -> &str { "my-router" }
}

Autoscaling

use forge_orchestration::AutoscalerConfig;

let config = AutoscalerConfig::default()
    .upscale_threshold(0.8)
    .downscale_threshold(0.3)
    .hysteresis_secs(300)
    .bounds(1, 100);

Storage

use forge_orchestration::storage::{MemoryStore, FileStore};

let memory = MemoryStore::new();
let file = FileStore::open("/var/lib/forge/state.json")?;

Metrics

use forge_orchestration::ForgeMetrics;

let metrics = ForgeMetrics::new()?;
metrics.record_job_submitted();
metrics.record_scale_event("my-job", "up");
let text = metrics.gather_text()?;

SDK Functions

Environment Variables

Builder Configuration

use forge_orchestration::ForgeBuilder;

ForgeBuilder::new()
    .with_nomad_api("http://localhost:4646")
    .with_nomad_token("secret-token")
    .with_store_path("/var/lib/forge/state.json")
    .with_node_name("forge-1")
    .with_datacenter("dc1")
    .with_autoscaler(AutoscalerConfig::default())
    .with_metrics(true)
    .build()?

License

Apache 2.0