grommet 0.1.0

Thread-per-core, key-affine work scheduling with hardware-aware placement
Documentation

Why "grommet"

A grommet is the little metal ring you press into a hole so the rope can pass through without sawing the canvas in half — and so the hole doesn't tear itself wider every time you pull.

That is the whole job here. One reinforced opening per core; work goes through it exactly once, from whoever submitted it to the shard that owns its key. On the far side nothing needs to be Send, nothing needs a lock, and a dispatched item holds the only copy of its state. Then you pull harder — and the mailbox fills, backpressure walks back to the submitter, and the ring keeps its shape instead of letting the load rip through the sheet.

Keeps things tight. Keeps things locked. Doesn't chafe.

How it works

Work carries an affine key. Every item for one key is handled by one shard, in submission order, one at a time — so the state behind that key needs no locking and no atomics, because while an item is being processed it holds the only copy. Shards are pinned to cores, each running a single-threaded runtime.

Within a shard, keys are dispatched round-robin from per-class ready rings. That bounds starvation strictly rather than statistically: a key at position k runs within k dispatches, no matter how much work a busier key has queued. Each class has its own in-flight budget, so saturating one — CPU-bound work, a slow dependency — cannot starve another.

impl Work for Job {
    type Key = u64;
    type Id = u128;
    fn key(&self) -> u64 { self.account }
    fn class(&self) -> ClassId { IO }
    fn request_id(&self) -> Option<u128> { Some(self.attempt) }
    fn time_to_live(&self) -> Option<Duration> { Some(Duration::from_millis(50)) }
}

impl Processor for Ledger {
    type Work = Call<Job, i64>;
    type State = i64;
    type Error = LedgerError;

    async fn process(&self, key: u64, state: Option<i64>, call: Call<Job, i64>)
        -> Result<Disposition<i64>, LedgerError>
    { /* you hold the only copy of this key's state */ }
}

let runtime = Runtime::<Ledger, _, 2>::builder(shards, [2048, 64])
    .pin(PinPolicy::Require)
    .coalesce_duplicates(true)
    .spawn(|shard| Ledger::new(shard))?;

let balance = runtime.router().call(job).await?;

Crates

Crate What it is
grommet-core The scheduler as a pure data structure. No async, no clock, no IO.
grommet The runtime: clock, traits, router, shard reactor, metrics.
grommet-topology Reads the machine — NUMA, SMT, P/E cores, cgroup quota — and plans where shards and offload workers go.
grommet-offload Pinned, bounded Rayon pools for CPU-bound work, one per memory node.
grommet-testkit Fault injection and conformance checks for your processor.
grommet-macros The assertion macros the scheduler's invariant checks are written in.
examples/accounts A worked example, and the proof the abstractions survive use.

grommet-core depends on ahash and nothing else. grommet adds tokio and futures. Reading the machine lives in grommet-topology, which wraps hwlocality and builds libhwloc from source by default — turn off its vendored feature to link a system one. Rayon is a separate crate you opt into.

What the runtime guarantees

  • One owner per key. A dispatched item receives the key's state by value. No other item for that key can be in flight, so there is nothing to lock.
  • Bounded starvation. Arrivals and completions both go to the back of a ring; dispatch pops the front. grommet-testkit measures the resulting wait and fails if it ever exceeds one rotation.
  • Independent class budgets. A flood of compute work fills the compute budget and ring only, leaving IO free to keep dispatching.
  • End-to-end backpressure. At the pending cap a shard stops admitting, its bounded mailbox fills, and Router::submit suspends the caller. try_submit sheds instead, handing the work back so you can answer your client.
  • Deadlines cost nothing to miss. Work past its deadline is discarded at dispatch, before it spends a turn.
  • Contained panics. A panicking process future cannot unwind the reactor. It is caught, counted, and the key's state is discarded so it reloads.
  • Safe eviction. A key is quiesced while on_evict flushes its state, so a write-back cannot race a reload of the same key.
  • Nothing is dropped on the way out. Closing the mailbox drains what is queued, and then every key still holding state is handed to on_evict before the shard exits. Resident state is a write-back cache; a shutdown that skipped it would lose writes the processor was told it could keep.

What it deliberately does not do

  • No work stealing. It would break the one-owner guarantee, which is the whole point. Rebalancing a skewed workload means migrating a key, which the router's slot table is built for.
  • No Send futures. Work is Send because it crosses once from submitter to shard. Nothing after that is. Rc and Cell are correct here, and code written against Send futures and work stealing will not fit. If you want that, use an ordinary multi-threaded executor.
  • No durable deduplication. In-flight coalescing suppresses a retry while its original is still outstanding. Once the original completes its id leaves the index, because answering a later retry correctly needs the original's recorded outcome — which only your store has. An in-memory dedup table would be lost on exactly the restart it exists to survive.
  • No replies unless you ask. Submission reports whether work was accepted, not what it produced. Wrap work in a Call for request/response; a reply channel costs an allocation and two atomics, and ingestion pipelines have no caller to answer.

Correctness contract for a processor

  • Returning Err always discards the key's resident state; classify it with Fallout::InDoubt when the durable outcome is unknown and Untouched when it definitely did not apply. A failure that leaves your state intact is not an error — return Ok(Disposition::Keep(state)).
  • Every mutation should carry a caller-stable request id, and a retry must reuse it.
  • Work::key, class and request_id are read once, at submission. The scheduler never asks again, so an inconsistent implementation cannot corrupt its rings.

Gates

just test       # formatting, strict Clippy, ordinary tests
just sim        # optimized deterministic simulation with fault injection
just miri       # undefined-behaviour check over the scheduler's unsafe slab
just coverage   # MC/DC instrumentation and a decision-layer line gate
just mutants    # assertion-strength check under the simulation configuration
just fuzz-list  # exact Bolero target names (run before fuzzing)
just fuzz TARGET        # coverage-guided model/fault fuzzing with ASan
just reduce TARGET      # replay and minimize one saved failure
just deny       # advisories, bans, and licenses
just bench      # domain, scheduler and full-reactor baselines

Safety-relevant modes travel through cfg, not Cargo features, because features unify across a dependency graph and a simulation build must never be reachable by accident. Three compile_error! tripwires reject leakage into an optimized build.

Unsafe code

There is exactly one unsafe module: the queue slab in grommet-core, which indexes without bounds checks. Every index it dereferences is one it allocated itself, never caller data, and that invariant is stated at the top of the file, debug_assert!ed at every use, and checked by a randomized model test against reference queues that runs under Miri in CI. Every other crate is #![deny(unsafe_code)].

Evidence domains

PostgreSQL and Redis adapters are outside mutation scoring and coverage until a conformance suite runs the same contract against real services. A green simulator cannot prove SQL, wire-protocol or pool-configuration truth. HTTP and gRPC are proven by Turmoil instead, across a simulated network that can be partitioned mid-request.

The example is part of the design

examples/accounts is a real account service: durable state behind an idempotency key, a non-authoritative cache, CPU-bound work that must not stall the reactor, and a commit whose acknowledgement can be lost. It exists because an abstract scheduler with no concrete user is how these designs acquire traits nobody can implement.

It is also where the interesting proofs live:

  • every_single_failure_position_reconciles_under_replay runs the whole stack once per injectable operation — 23 of them — and insists each one converges on the same durable state under replay.
  • a_partition_and_an_in_doubt_retry_cross_the_whole_stack loses a commit's acknowledgement, partitions the client mid-retry, repairs the network, and checks the replay is recognised as a duplicate rather than applied twice.
  • an_old_duplicate_cannot_regress_newer_state covers the subtle one: a duplicate carries the balance recorded for its id, which may be older than what is resident. Letting it overwrite would livelock a genuinely missing request behind version conflicts forever.

Performance

Machine-local evidence from 2026-08-13, not portable SLOs. Use Criterion's named baselines on the same quiet machine for optimization decisions.

Workload Median
Routing hash 1.15 ns
ULID creation 41.2 ns
Scheduler admit + dispatch + complete, hot key 14.7 ns
Scheduler admit + dispatch + complete, 100k keys 21.9 ns
Pure revalue kernel, one 200k-iteration scenario 429.6 µs
One shard, 64 concurrent reads, admit_batch = 1 38.2 µs
One shard, 64 concurrent reads, admit_batch = 64 33.3 µs
One shard, sequential reads on one hot key 5.82 µs

Batching admission amortizes cross-thread wakeups, which dominate at high rates: 64 is where the curve flattens, and it is the default. The scheduler triple was 27.4 ns before per-key queues became intrusive lists over a shared slab; that is the same machine and the same operation, but it was not a controlled A/B, so treat it as suggestive.

License

Licensed under either of

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.