# windows-ioring-sys
Memory-safe Rust over the Windows `IoRing` submission/completion ring.
**Windows only.** Every item is behind `cfg(windows)`; the crate builds to an
empty shell on other platforms.
Windows 11 and Server 2022 added `IoRing`, a submission/completion ring for file
I/O closer in shape to `io_uring` than to anything else Windows offers. This
crate raises those primitives into safe Rust. The ring holds each in-flight
operation's buffer on the caller's behalf and hands it back at the pop that
observes its completion, so no buffer the kernel may still be using is reachable
from safe code. The tracking that makes this possible has a cost, which may
allocate: an entry in a hash set for every operation in flight, and for an owned
push an entry in a hash map holding its payload
([D-78](DESIGN-NOTES.md#d-78)).
## Example
Submit a read, then pop its completion once it is ready:
```rust,no_run
use windows_ioring_sys::{Batch, IoRing, SharedHandle};
use std::os::windows::io::OwnedHandle;
let file = std::fs::File::open(r"C:\some\file.bin")?;
let shared = SharedHandle::new(OwnedHandle::from(file));
// The ring holds each operation's buffer, so `T` says what it holds.
let mut ring = IoRing::<Vec<u8>>::with_inventory(8, 8)?;
{
let mut batch = Batch::new(&mut ring);
batch.read_owned(&shared, vec![0_u8; 4096], (), 0, Default::default())?;
batch.submit_and_wait(1, 5_000)?;
}
// The pop hands the buffer back. There is nothing to hold onto in between,
// and nothing to match against: the ring already knows which operation this
// completion belongs to.
// `pop_within` rather than `try_pop`: `submit_and_wait` returning does not
// promise a completion is already poppable. RS-P-5 leaves the kernel free to
// post it later, so unwrapping `try_pop` here would assert a guarantee this
// crate does not make -- the same shape its own tests refuse at the source.
let (completion, held) = ring
.pop_within(std::time::Duration::from_secs(5))?
.expect("a completion arrives within the bound");
completion.result()?;
let (buffer, ()) = held.expect("the ring was holding this read's buffer");
println!("read {} bytes", buffer.expect("a read carries a buffer").len());
# Ok::<(), std::io::Error>(())
```
`EventDelivery` wires completions to the thread pool instead, so no thread
ever calls `try_pop` itself; see `examples/model_a_delivery.rs` for that shape,
and the "Choosing a delivery architecture" section below for when to reach for
each.
## Scope: a file data plane, not a general completion backend
The kernel's operation table is fixed at seven entries -- no-op, read, write,
flush, register-files, register-buffers, and cancel. There is no ioctl operation,
no socket operation, and no directory-change operation, and unlike Linux's
`io_uring` -- which grew to roughly fifty opcodes including full socket support
-- Windows `IoRing` has not grown beyond file I/O.
So this crate does not replace
[`windows-overlapped-io-sys`](../windows-overlapped-io-sys/README.md), which
remains the crate for arbitrary I/O on arbitrary handles. Use this one for
high-volume file reads and writes; use that one for everything else. Neither can
subsume the other, and the division is the kernel's rather than this
repository's.
## Availability is a runtime question
`IoRing` ships with Windows 11 and Server 2022, but three separate facts about it
are decided at runtime rather than at compile time:
- which ring version the system supports;
- whether the ring is a real kernel ring or a **user-mode emulation** with no
kernel benefit;
- whether the completion-event feature the thread-pool delivery path depends on
is available at all.
All three are answerable without creating a ring, and this crate surfaces them
rather than hiding them -- a consumer reaching for a ring to maximize throughput
needs to know if they are getting an emulation.
## Choosing a delivery architecture
There are two coherent high-performance shapes, and they are mutually exclusive
on the hot path. Picking the wrong one costs more than any API detail in this
crate, so the trade-off is documented in full under "Two delivery architectures"
in [DESIGN-NOTES.md](DESIGN-NOTES.md), including why the NUMA node is the wrong
key for partitioning rings and why buffer placement likely matters more than
thread placement.
**Model A, threadless dispatch -- shared queue, kernel load-balances.**
`EventDelivery` wires the
ring's completion event to a `ThreadpoolWait` from `windows-threadpool-sys`, so
no thread of yours ever blocks on I/O. Start here; see
[examples/model_a_delivery.rs](examples/model_a_delivery.rs) for a full worked
example that submits without waiting and receives every completion on a pool
thread. A caller that keeps its ring behind its own lock takes `CompletionWake`
instead: the pool only tells it when to drain, and it drains its own ring to
empty. `EventDelivery` is built on it, so the two share one delivery protocol
([D-84](DESIGN-NOTES.md#d-84)).
**Model B, dedicated domain thread -- shared-nothing execution domains.** One
pinned thread per domain,
owning its ring, its buffer pool, and its shard of the work, parked directly in
`Batch::submit_and_wait` -- the fused submit-and-wait *is* the event loop. This
is the shape `IoRing`'s own API is built for.
**Model B's wakeup source is separable from Model B's identity.** What makes it
Model B is who owns, submits, and drains -- one pinned thread, no sharing on the
data path -- not what that thread blocks on. There are two answers for the
latter, and both are Model B:
- **Fused submit-and-wait**, blocking in `Batch::submit_and_wait`. Use it when
the domain's only I/O is ring I/O: nothing to re-arm, nothing to multiplex.
- **A multiplexed wait**, blocking in `WaitForMultipleObjects` over
`IoRing::completion_event` alongside other handles. Use it when the domain
must also service a shutdown event, a socket, an overlapped operation, or a
timer. `IoRing::completion_event` hands back an owned *duplicate* of the
ring's event, so the caller keeps its ring. See
[examples/model_b_multiplexed.rs](examples/model_b_multiplexed.rs) for the
whole shape end to end, including shutdown with I/O still outstanding.
Picking the second is not "Model A with extra steps" and costs none of the
locality that motivated Model B. It does inherit one contract: the ring's event
is **edge-triggered on the completion queue going empty to non-empty**, so a
waiter must drain to empty before waiting again, on every pass, and must treat a
wake with nothing to pop as normal. That is measured rather than documented by
Win32, and getting it wrong hangs rather than merely slows -- read
`IoRing::completion_event`'s own docs before using it.
**The `drain_preceding` barrier stops at the ring's edge.** This is the reason
the multiplexed shape has to exist. `IOSQE_FLAGS_DRAIN_PRECEDING_OPS` orders
SQEs against SQEs and is powerless in both directions across the ring boundary:
it can neither make a ring operation wait for an overlapped one nor make an
overlapped operation wait for ring operations. A consumer mixing ring and
non-ring I/O -- the normal case, not an exotic one -- must therefore enforce
that ordering in its own code, and the multiplexed wait is what lets it do so
without surrendering the ring or parking a thread in a blocking drain. The
barrier is also ring-wide in *reach*: a drained flush waits on every operation
outstanding on the ring when it is reached, not only the ones a caller cares
about, so it takes as long as the slowest of them. It is one-sided, though --
operations pushed after it are not held back and can complete first. See
`PushOptions::drain_preceding` and [D-47](DESIGN-NOTES.md#d-47-detail).
Most real applications want Model B on the hot data path and Model A everywhere
else -- the control plane, background work, cold paths -- where the thread
pool's quiescence is worth more than locality. This crate supports both as
first-class; neither is a degraded form of the other.
## Durability
Three facts, all measured rather than documented by Win32, and all of them
things a consumer gets wrong by default. `Batch::flush_owned`, `FlushCoverage`,
`WriteCaching` and `FlushMode` state them in full; this is the summary that
stops a reader from never looking.
1. **There is no FUA.** `BuildIoRingWriteFile`'s entire flag set is
`{FILE_WRITE_FLAGS_NONE, FILE_WRITE_FLAGS_WRITE_THROUGH}`, and write-through
is a cache directive to the OS, not a device-level guarantee -- whether it
becomes a Force Unit Access bit depends on the driver, the volume, and the
device's write-cache setting. A completed write-through write may still be
sitting in a volatile device cache.
2. **The flush operation is the only durability primitive the ring has**, and
only in a mode that syncs the device -- `FlushMode::NoSync` deliberately
does not, which makes it the one mode that commits nothing.
3. **A flush without the barrier covers nothing.** An unflagged flush is an
ordinary operation competing with the writes before it, and it frequently
wins, so its completion proves nothing about them. This is why
`Batch::flush_owned` requires a `FlushCoverage` rather than defaulting: the
obvious spelling was a silent data-loss bug, invisible until power is lost.
Note that seeing your flush land last on your hardware is not evidence you
can omit the barrier -- which direction the reordering shows in is
device-dependent.
So **durability is a property of an epoch, never of an individual write**,
because the ring offers no per-write primitive to make it one: stream the
writes, close the epoch with one covering flush, and wait on the flush rather
than on the writes. The barrier that makes this correct waits on everything
outstanding on the ring, so the correct construction is also the expensive one. "Durability on
the ring" in [DESIGN-NOTES.md](DESIGN-NOTES.md) has the full shape and the three
ways to pay for it.
[examples/epoch_log/](examples/epoch_log/) builds all of that as a running
program: a write-ahead log with a written-down durability contract, group
commit, a multiplexed wait, a thread-pool control plane, replay with a negative
control, and all three commit strategies measured against each other on your
machine. It needs the `threadpool` feature, and it is a **demonstration of a
pattern, not supported API surface** -- see the caveat under
[Cargo features](#cargo-features) below.
## Cargo features
| `threadpool` | on | `EventDelivery` and `CompletionWake` (Model A), and with them the dependency on `windows-threadpool-sys`. |
`EventDelivery` and `CompletionWake` are the only items in this crate that need a thread pool, so a
Model B consumer -- one pinned thread per domain, parked in
`Batch::submit_and_wait`, owning its own ring -- otherwise links a dependency it
never calls. Turning the feature off drops that dependency entirely:
```toml
[dependencies]
windows-ioring-sys = { version = "0.1", default-features = false }
```
The gate is justified on layering rather than runtime cost: linking
`windows-threadpool-sys` creates no threads, because the Win32 default pool is a
process-wide facility instantiated lazily on first use. A ring wrapper simply
does not intrinsically depend on a thread pool. Default-on keeps the change
additive, so no existing consumer has to do anything.
### The examples are demonstrations, not API
`examples/` is where this crate shows compositions it deliberately does not
ship. [examples/epoch_log/](examples/epoch_log/) is the largest: it makes
policy choices -- what an epoch is, when to commit, what durability is reported
as, which of the three commit strategies to pay for -- that the crate itself
refuses to make, because they depend on a workload the crate cannot see (D-8,
D-26).
So treat it as a worked answer to "how do these primitives go together", not as
a component. **Nothing under `examples/` is public API, none of it is covered
by this crate's semantic versioning, and vendoring it makes its policy choices
yours to maintain.** If you copy it, copy it as a starting point you own.
## Topology guidance
This crate does not partition anything for you (D-8): it makes a ring cheap and
correct, makes its affinity explicit, and leaves sizing a Model B execution
domain to the caller.
- **Size a domain by the outermost cache level that partitions the machine, not by NUMA node.** Node count is a
firmware setting a process cannot see, and most real deployments are
virtualized, where NUMA topology is often invisible entirely. Ask
`outermost_partitioning_cache()` rather than filtering on `CacheLevel == 3`:
a shipping ARM part reports no L3 at all, and a machine can report an L3
spanning every processor above a real L2 partition, where a level filter
returns one whole-machine domain and calls it a cache-aware partition. See
[examples/cache_domains.rs](examples/cache_domains.rs) for a runnable
enumeration, built on the safe `GetLogicalProcessorInformationEx` wrapper in
[`windows-topology-sys`](../windows-topology-sys/README.md).
- **Processor groups are a hard floor.** A thread's affinity is a
`GROUP_AFFINITY` and a ring's waiter lives in exactly one group, so above 64
logical processors the partition is forced whether or not it is wanted.
- **Buffer placement likely dominates thread placement.** `VirtualAllocExNuma`
on the node closest to the device, registered once into that domain's ring
via `Batch::register_buffers`, is very likely the highest-leverage locality
decision available -- independent of everything above about completion
routing. `NumaBuffer` is that allocation; *which* node is still the caller's
answer, and this crate does not guess it.
`examples/ring_copy` is where these three points become runnable policy: it
copies one file to another through per-domain rings, sized by a named
`ByCache`/`ByNode`/`ByPackage`/`ByCore`/`Single` policy, with placed buffers and a
`--placement local|remote` switch to make the placement effect measurable. It
is a **sample**, not library surface -- this crate itself depends on no
partitioning policy and does not depend on `windows-topology-sys`; only the
sample does.
## License
MIT. Copyright (c) Mike Grier.