rime

Lock-free delta-xDS control plane for Envoy, written in Rust.

rime is a transport-agnostic xDS management server kernel. It tracks Envoy resources (clusters, listeners, endpoints, routes), maintains a persistent content-addressed hash index, and delivers incremental delta xDS updates — sending only what changed, not everything, every time.

The xDS control plane problem has three hard parts: avoiding lock contention on the hot read path, shipping only the changed resources to each subscriber, and tracking per-subscriber acknowledgements so stale state never accumulates. rime solves all three with four small, composable primitives and zero Arc<Mutex<T>>.

Why delta xDS?

Envoy supports two xDS modes: state-of-the-world (SotW) and delta (incremental). Most control plane libraries — including go-control-plane — implement only SotW. When a single pod restarts in a 200-service cluster, SotW rebuilds and transmits all 200 resources to every subscriber. Delta sends one.

N = 200 services, 1 resource changed:

  SotW (go-control-plane path)  11,819 ns  O(N)
  delta_naive                   24,141 ns  O(N) + hashing overhead
  delta_persistent (rime)          757 ns  O(1) — index maintained between events

The key insight: a naive delta that re-hashes all N resources to find the one that changed is still O(N). A persistent hash index maintained across events reduces the per-update cost to a single hash and a map lookup — constant time regardless of cluster size.

Crossover point: delta is faster than SotW at N ≥ 25 services. At N = 200 it is 15.6× faster. At N = 1000 the gap is the same: ~760 ns vs O(N) growing without bound.

Architecture

┌─────────────────────────────────────────────────────┐
│  Plane  (single writer, &mut self API)               │
│                                                      │
│  ┌──────────┐   ┌─────────────────────────────────┐ │
│  │  Index   │   │  Snapshot (ArcSwap<ResourceMap>) │ │
│  │  xxh3    │   │  lock-free atomic pointer swap   │ │
│  │  HashMap │   │  pub(crate) store, pub load      │ │
│  └──────────┘   └─────────────────────────────────┘ │
│         │                      │                     │
│         └────── apply/remove ──┘                     │
│                                │                     │
│                    watch::Sender  ──── notifies ─►   │
└─────────────────────────────────────────────────────┘
                                          │
                          ┌───────────────▼────────────┐
                          │  Server  (fan-out factory)  │
                          │                             │
                          │  subscribe(rx) → Subscriber │
                          └─────────────────────────────┘
                                          │
                        ┌─────────────────▼──────────────────┐
                        │  Subscriber  (per-connection state) │
                        │                                     │
                        │  acked: HashMap<String, u64>        │
                        │  initial() → full Update            │
                        │  changed() → incremental Update     │
                        │  acknowledge(name, hash)            │
                        └─────────────────────────────────────┘

Plane is the single writer. &mut self on every mutating method is a compile-time single-writer guarantee with no runtime cost.

Index is a persistent HashMap<String, u64> mapping resource name → xxh3 hash. It never allocates on the unchanged fast path: get() borrows by &str, insert() fires only when content actually differs.

Snapshot is an ArcSwap<ResourceMap>. load() is one atomic instruction. store() is pub(crate) — only Plane can write; external callers receive Arc<Snapshot> with read-only access.

Subscriber computes its delta from the current snapshot against its own acked map. Each subscriber tracks independently. No subscriber state leaks to any other.

Quick start

[dependencies]
rime-xds = "0.1"
tokio    = { version = "1", features = ["sync"] }
bytes    = "1"

use rime::{Plane, Server};
use bytes::Bytes;

// Plane is the single writer. Server holds the snapshot for readers.
let (mut plane, rx) = Plane::new();
let server = Server::new(plane.snapshot());

// Populate initial state.
plane.apply("cluster/backend", Bytes::from_static(b"v1-proto-bytes"));
plane.apply("listener/ingress", Bytes::from_static(b"v1-proto-bytes"));

// Per-connection: subscribe and send the full initial snapshot.
let mut sub = server.subscribe(rx.clone());
let initial = sub.initial();
for (name, bytes) in &initial.resources {
    // send to Envoy via gRPC / DeltaDiscoveryResponse
    println!("send {} ({} bytes)", name, bytes.len());
}

// When Envoy ACKs a resource, record it so rime won't re-send.
sub.acknowledge("cluster/backend", xxhash_rust::xxh3::xxh3_64(b"v1-proto-bytes"));

// Later: a pod restarts. Apply the changed resource.
plane.apply("cluster/backend", Bytes::from_static(b"v2-proto-bytes"));

// Subscriber wakes only when the snapshot changes, then diffs against acked state.
if let Some(update) = sub.changed().await {
    assert_eq!(update.resources.len(), 1);        // only cluster/backend
    assert_eq!(update.resources[0].0, "cluster/backend");
    // listener/ingress was ACKed and unchanged — never transmitted
}

// Resource removed from the control plane.
plane.remove("listener/ingress");
if let Some(update) = sub.changed().await {
    assert!(update.resources.is_empty());
    assert_eq!(update.removed, ["listener/ingress"]);  // xDS tombstone
    sub.acknowledge_removed("listener/ingress");
}

Benchmarks

All numbers are Criterion means on an AMD Ryzen 9 (Linux, single-thread). Raw results are in benches/.

H1 — Lock-free snapshot reads (ArcSwap vs RwLock)

ArcSwap is immune to write pressure: load() is one atomic instruction. RwLock performance degrades under real write contention — these numbers are single-threaded; contended numbers are considerably worse.

H2 — Zero-copy fan-out (Bytes vs Vec<u8>)

At 100 subscribers, each needing a handle to a 10 KB resource:

Bytes::clone() is a 4.2 ns atomic refcount increment at any payload size. Vec<u8>::clone() at 10 KB costs 22× more per handle.

H3 — Content-addressed versioning (xxh3 vs alternatives)

At 1 KB (typical xDS resource, e.g., a Cluster or Listener):

Content-addressed versioning with xxh3 costs 36.7 ns per 1 KB resource — less than 1% overhead relative to the serialization cost it replaces. A reconnecting Envoy instance sends its known hashes; rime sends only resources whose hash differs. No monotonic integer versions, no version skew.

H4 — Delta update cost vs state-of-the-world rebuild

1 resource changed, rest identical. "persistent" = index maintained across events (the production pattern: only re-hash the resource you know changed).

Persistent delta is flat at ~760–775 ns regardless of cluster size. SotW grows linearly: crossover is at N ≈ 20 services. At N = 200, persistent delta is 15.6× faster than SotW.

Comparison with go-control-plane

go-control-plane is a mature, production-proven library. rime is the alternative when you need incremental xDS updates, zero-copy fan-out, and Rust's ownership model to enforce invariants the compiler can verify.

xDS resource types

rime stores serialized xDS resources as opaque Bytes. It is agnostic to which of the following types a given name refers to — callers handle serialization:

A typical xDS management server wraps rime::Plane + rime::Server and drives them from a Kubernetes informer or similar watch mechanism, serializing resources with prost or tonic before calling plane.apply().

xDS resource types and the DeltaDiscoveryService wire protocol

rime's Subscriber::initial() and Subscriber::changed() map directly onto the DeltaDiscoveryService wire protocol:

• initial() → populate DeltaDiscoveryResponse.resources for the first response
• changed() → populate subsequent incremental DeltaDiscoveryResponse.resources
• Update::removed → populate DeltaDiscoveryResponse.removed_resources
• acknowledge(name, version_info) → called when DeltaDiscoveryRequest.response_nonce confirms receipt

The nonce / version bookkeeping is the caller's responsibility; rime tracks content hashes, not nonces.

Prior art

• go-control-plane — the official Envoy xDS library in Go. SotW only; no delta xDS. Uses sync.RWMutex.
• xds-rs — Rust xDS type bindings (generated protobuf). No control plane logic.
• envoy-control — JVM (Kotlin/Spring), SotW.

None of the Rust crates on crates.io implement a delta xDS control plane or a persistent content-addressed hash index. rime is the first.

License

Licensed under either of MIT or Apache 2.0 at your option.