windows-ioring-sys

Memory-safe Rust over the Windows IoRing submission/completion ring.

Windows only. Every item is behind cfg(windows); the crate builds to an empty shell on other platforms.

Windows 11 and Server 2022 added IoRing, a submission/completion ring for file I/O closer in shape to io_uring than to anything else Windows offers. This crate raises those primitives into safe Rust. The ring holds each in-flight operation's buffer on the caller's behalf and hands it back at the pop that observes its completion, so no buffer the kernel may still be using is reachable from safe code. The tracking that makes this possible has a cost, which may allocate: an entry in a hash set for every operation in flight, and for an owned push an entry in a hash map holding its payload (D-78).

Example

Submit a read, then pop its completion once it is ready:

use windows_ioring_sys::{Batch, IoRing, SharedHandle};
use std::os::windows::io::OwnedHandle;

let file = std::fs::File::open(r"C:\some\file.bin")?;
let shared = SharedHandle::new(OwnedHandle::from(file));
// The ring holds each operation's buffer, so `T` says what it holds.
let mut ring = IoRing::<Vec<u8>>::with_inventory(8, 8)?;

{
    let mut batch = Batch::new(&mut ring);
    batch.read_owned(&shared, vec![0_u8; 4096], (), 0, Default::default())?;
    batch.submit_and_wait(1, 5_000)?;
}

// The pop hands the buffer back. There is nothing to hold onto in between,
// and nothing to match against: the ring already knows which operation this
// completion belongs to.
// `pop_within` rather than `try_pop`: `submit_and_wait` returning does not
// promise a completion is already poppable. RS-P-5 leaves the kernel free to
// post it later, so unwrapping `try_pop` here would assert a guarantee this
// crate does not make -- the same shape its own tests refuse at the source.
let (completion, held) = ring
    .pop_within(std::time::Duration::from_secs(5))?
    .expect("a completion arrives within the bound");
completion.result()?;
let (buffer, ()) = held.expect("the ring was holding this read's buffer");
println!("read {} bytes", buffer.expect("a read carries a buffer").len());
# Ok::<(), std::io::Error>(())

EventDelivery wires completions to the thread pool instead, so no thread ever calls try_pop itself; see examples/model_a_delivery.rs for that shape, and the "Choosing a delivery architecture" section below for when to reach for each.

Scope: a file data plane, not a general completion backend

The kernel's operation table is fixed at seven entries -- no-op, read, write, flush, register-files, register-buffers, and cancel. There is no ioctl operation, no socket operation, and no directory-change operation, and unlike Linux's io_uring -- which grew to roughly fifty opcodes including full socket support -- Windows IoRing has not grown beyond file I/O.

So this crate does not replace windows-overlapped-io-sys, which remains the crate for arbitrary I/O on arbitrary handles. Use this one for high-volume file reads and writes; use that one for everything else. Neither can subsume the other, and the division is the kernel's rather than this repository's.

Availability is a runtime question

IoRing ships with Windows 11 and Server 2022, but three separate facts about it are decided at runtime rather than at compile time:

• which ring version the system supports;
• whether the ring is a real kernel ring or a user-mode emulation with no kernel benefit;
• whether the completion-event feature the thread-pool delivery path depends on is available at all.

All three are answerable without creating a ring, and this crate surfaces them rather than hiding them -- a consumer reaching for a ring to maximize throughput needs to know if they are getting an emulation.

Choosing a delivery architecture

There are two coherent high-performance shapes, and they are mutually exclusive on the hot path. Picking the wrong one costs more than any API detail in this crate, so the trade-off is documented in full under "Two delivery architectures" in DESIGN-NOTES.md, including why the NUMA node is the wrong key for partitioning rings and why buffer placement likely matters more than thread placement.

Model A, threadless dispatch -- shared queue, kernel load-balances. EventDelivery wires the ring's completion event to a ThreadpoolWait from windows-threadpool-sys, so no thread of yours ever blocks on I/O. Start here; see examples/model_a_delivery.rs for a full worked example that submits without waiting and receives every completion on a pool thread. A caller that keeps its ring behind its own lock takes CompletionWake instead: the pool only tells it when to drain, and it drains its own ring to empty. EventDelivery is built on it, so the two share one delivery protocol (D-84).

Model B, dedicated domain thread -- shared-nothing execution domains. One pinned thread per domain, owning its ring, its buffer pool, and its shard of the work, parked directly in Batch::submit_and_wait -- the fused submit-and-wait is the event loop. This is the shape IoRing's own API is built for.

Model B's wakeup source is separable from Model B's identity. What makes it Model B is who owns, submits, and drains -- one pinned thread, no sharing on the data path -- not what that thread blocks on. There are two answers for the latter, and both are Model B:

• Fused submit-and-wait, blocking in Batch::submit_and_wait. Use it when the domain's only I/O is ring I/O: nothing to re-arm, nothing to multiplex.
• A multiplexed wait, blocking in WaitForMultipleObjects over IoRing::completion_event alongside other handles. Use it when the domain must also service a shutdown event, a socket, an overlapped operation, or a timer. IoRing::completion_event hands back an owned duplicate of the ring's event, so the caller keeps its ring. See examples/model_b_multiplexed.rs for the whole shape end to end, including shutdown with I/O still outstanding.

Picking the second is not "Model A with extra steps" and costs none of the locality that motivated Model B. It does inherit one contract: the ring's event is edge-triggered on the completion queue going empty to non-empty, so a waiter must drain to empty before waiting again, on every pass, and must treat a wake with nothing to pop as normal. That is measured rather than documented by Win32, and getting it wrong hangs rather than merely slows -- read IoRing::completion_event's own docs before using it.

The drain_preceding barrier stops at the ring's edge. This is the reason the multiplexed shape has to exist. IOSQE_FLAGS_DRAIN_PRECEDING_OPS orders SQEs against SQEs and is powerless in both directions across the ring boundary: it can neither make a ring operation wait for an overlapped one nor make an overlapped operation wait for ring operations. A consumer mixing ring and non-ring I/O -- the normal case, not an exotic one -- must therefore enforce that ordering in its own code, and the multiplexed wait is what lets it do so without surrendering the ring or parking a thread in a blocking drain. The barrier is also ring-wide in reach: a drained flush waits on every operation outstanding on the ring when it is reached, not only the ones a caller cares about, so it takes as long as the slowest of them. It is one-sided, though -- operations pushed after it are not held back and can complete first. See PushOptions::drain_preceding and D-47.

Most real applications want Model B on the hot data path and Model A everywhere else -- the control plane, background work, cold paths -- where the thread pool's quiescence is worth more than locality. This crate supports both as first-class; neither is a degraded form of the other.

Durability

Three facts, all measured rather than documented by Win32, and all of them things a consumer gets wrong by default. Batch::flush_owned, FlushCoverage, WriteCaching and FlushMode state them in full; this is the summary that stops a reader from never looking.

1. There is no FUA. BuildIoRingWriteFile's entire flag set is {FILE_WRITE_FLAGS_NONE, FILE_WRITE_FLAGS_WRITE_THROUGH}, and write-through is a cache directive to the OS, not a device-level guarantee -- whether it becomes a Force Unit Access bit depends on the driver, the volume, and the device's write-cache setting. A completed write-through write may still be sitting in a volatile device cache.
2. The flush operation is the only durability primitive the ring has, and only in a mode that syncs the device -- FlushMode::NoSync deliberately does not, which makes it the one mode that commits nothing.
3. A flush without the barrier covers nothing. An unflagged flush is an ordinary operation competing with the writes before it, and it frequently wins, so its completion proves nothing about them. This is why Batch::flush_owned requires a FlushCoverage rather than defaulting: the obvious spelling was a silent data-loss bug, invisible until power is lost. Note that seeing your flush land last on your hardware is not evidence you can omit the barrier -- which direction the reordering shows in is device-dependent.

So durability is a property of an epoch, never of an individual write, because the ring offers no per-write primitive to make it one: stream the writes, close the epoch with one covering flush, and wait on the flush rather than on the writes. The barrier that makes this correct waits on everything outstanding on the ring, so the correct construction is also the expensive one. "Durability on the ring" in DESIGN-NOTES.md has the full shape and the three ways to pay for it.

examples/epoch_log/ builds all of that as a running program: a write-ahead log with a written-down durability contract, group commit, a multiplexed wait, a thread-pool control plane, replay with a negative control, and all three commit strategies measured against each other on your machine. It needs the threadpool feature, and it is a demonstration of a pattern, not supported API surface -- see the caveat under Cargo features below.

Cargo features

EventDelivery and CompletionWake are the only items in this crate that need a thread pool, so a Model B consumer -- one pinned thread per domain, parked in Batch::submit_and_wait, owning its own ring -- otherwise links a dependency it never calls. Turning the feature off drops that dependency entirely:

[dependencies]
windows-ioring-sys = { version = "0.1", default-features = false }

The gate is justified on layering rather than runtime cost: linking windows-threadpool-sys creates no threads, because the Win32 default pool is a process-wide facility instantiated lazily on first use. A ring wrapper simply does not intrinsically depend on a thread pool. Default-on keeps the change additive, so no existing consumer has to do anything.

With the threadpool feature, EventDelivery::new and CompletionWake::new take windows-threadpool-sys's CallbackEnviron, so that crate's types are part of this crate's public API. A minor release of windows-threadpool-sys that changes those types is a minor release of this crate too, in the 0.x series: 0.8 of this crate pairs with 0.3 of windows-threadpool-sys.

The examples are demonstrations, not API

examples/ is where this crate shows compositions it deliberately does not ship. examples/epoch_log/ is the largest: it makes policy choices -- what an epoch is, when to commit, what durability is reported as, which of the three commit strategies to pay for -- that the crate itself refuses to make, because they depend on a workload the crate cannot see (D-8, D-26).

So treat it as a worked answer to "how do these primitives go together", not as a component. Nothing under examples/ is public API, none of it is covered by this crate's semantic versioning, and vendoring it makes its policy choices yours to maintain. If you copy it, copy it as a starting point you own.

Topology guidance

This crate does not partition anything for you (D-8): it makes a ring cheap and correct, makes its affinity explicit, and leaves sizing a Model B execution domain to the caller.

• Size a domain by the outermost cache level that partitions the machine, not by NUMA node. Node count is a firmware setting a process cannot see, and most real deployments are virtualized, where NUMA topology is often invisible entirely. Ask outermost_partitioning_cache() rather than filtering on CacheLevel == 3: a shipping ARM part reports no L3 at all, and a machine can report an L3 spanning every processor above a real L2 partition, where a level filter returns one whole-machine domain and calls it a cache-aware partition. See examples/cache_domains.rs for a runnable enumeration, built on the safe GetLogicalProcessorInformationEx wrapper in windows-topology-sys.
• Processor groups are a hard floor. A thread's affinity is a GROUP_AFFINITY and a ring's waiter lives in exactly one group, so above 64 logical processors the partition is forced whether or not it is wanted.
• Buffer placement likely dominates thread placement. VirtualAllocExNuma on the node closest to the device, registered once into that domain's ring via Batch::register_buffers, is very likely the highest-leverage locality decision available -- independent of everything above about completion routing. NumaBuffer is that allocation; which node is still the caller's answer, and this crate does not guess it.

examples/ring_copy is where these three points become runnable policy: it copies one file to another through per-domain rings, sized by a named ByCache/ByNode/ByPackage/ByCore/Single policy, with placed buffers and a --placement local|remote switch to make the placement effect measurable. It is a sample, not library surface -- this crate itself depends on no partitioning policy and does not depend on windows-topology-sys; only the sample does.

License

MIT. Copyright (c) Mike Grier.