gpu-async-reactor

Await GPU work as a memory descriptor. A GPU stream stores a sequence number into a word of host-visible memory behind its work. That word is to device completion what a file descriptor is to I/O readiness: this crate makes it a first-class awaitable source, and supplies the wait plans, the deadline estimator and a small executor around it, so a host thread can sleep until the GPU is nearly done, park in hardware on the completion word, and run other ready work in between.

📑 Contents

• 🚀 Quick start
• 🧭 Design
• 🔌 Backends and hosts
• ⚙️ Choosing an executor
• 📊 Benchmarks
• 🛡️ Safety
• ❗ Errors
• 🦀 MSRV
• 🚧 Status
• ⚖️ License

🚀 Quick start

[dependencies]
gpu-async-reactor = { version = "0.1", features = ["cuda"] }

Wait for a CUDA stream with the plan the Metrale serve A/B picked (park on the completion word, query the stream every millisecond as the error authority):

use gpu_async_reactor::backend::cuda::{CuStream, CudaPort, MappedLine};
use gpu_async_reactor::backend::wordwait::WordWait;
use gpu_async_reactor::host::std_host::StdClock;
use gpu_async_reactor::{Slot, WaitPlan, run_inline, wait};

let plan = WaitPlan::parse("flag:park_us=100,backstop_us=1000")?;
let line = MappedLine::new()?;                       // one 64-byte line, device-mapped
let stream = unsafe { CuStream::from_raw(raw) };     // your live CUstream
let (clock, mut seq) = (StdClock::new(), 0);
// ... enqueue work on `stream` ...
let mut port = CudaPort::new(&line, stream, Slot::new(0).unwrap(), &mut seq, WordWait::probe().ok());
let report = run_inline(wait(&plan, None, &mut port, &clock, &clock), "my wait")??;
println!("{:?} after {} ns, {} parks", report.kind, report.wait_ns, report.parks);

Run several waits and CPU work on one thread (no GPU needed for this one):

cargo run --example fill_executor --features std,wordwait
cargo run --release --example cuda_wait --features cuda      # needs an NVIDIA GPU

🧭 Design

One crate, organised as ports and hosts (SBIO): the core holds every decision and does no I/O; backends and hosts are generic parameters chosen once by the binary. There is no dyn in the core.

The wait, step by step:

block?  ── yes ──▶ port.block()                                   ▶ Blocked
  │ no
query() ── done ─▶                                                 ▶ Immediate (exact residual)
  │ pending
flag? ── arm(): enqueue store(seq) behind the work
predict? ── Timer.sleep_until(start + predicted − margin); query() ── done ─▶ Late (censored)
park: flag ── park(park_us) until the word reaches seq, query() every backstop_us
      poll ── query() until done                                   ▶ Parked (exact residual)

🔌 Backends

Every vendor entry point is resolved by name at run time (libcuda.so.1, libamdhip64.so), so the crate links nothing and a machine without the driver fails only the constructor. Adding hardware means one LineDriver table (the CUDA and HIP lines share the ownership logic) or one CompletionPort.

⚙️ Choosing an executor

resolve(requested, strictness, &caps) walks the chain once at startup. With Strictness::Fallback an unavailable executor yields the fallback plus a coded ExecutorFallback warning naming what was asked, why it is unavailable, and what runs; with Strictness::Strict it is an ExecutorRefused error. Declared is the generic shape: implement it for any other setting with a fallback chain.

📊 Benchmarks

All numbers are from the Metrale Engine on an NVIDIA GB10 (Grace Arm + Blackwell, coherent memory), CUDA 13, serving through the in-tree predecessor of this crate (identical logic). Method: same-session, rotated boots per arm, clean box (load < 4, no other GPU work), ratios paired to the spin baseline booted in the same rotation, 95 % CIs (Student-t on paired ratios); GPU energy from the NVML total-energy counter read twice per cell.

Wake latency and park cost (micro):

Serve A/B, Qwen3.6-35B-A3B FP8, n = 5 rotated boots, ratio to the spinning cuStreamSynchronize:

Reading: parking on the word holds throughput parity with a spin; adding the predicted sleep gives back two thirds of the scheduler core at C4-C16 for a 0.2-0.5 % throughput cost, and is at parity from C32. Waking from an interrupt or a file descriptor instead lost 0-3 % (35B) and up to 9 % (27B, C8) and multiplied context switches 9-50x per token. CPU power of a WFE park is not on any GB10 rail; only the GPU rail is metered. The executor A/B (naive vs gpu-reactor vs io-uring, same method, C1-C64) measured them equal within noise in tok/s, CPU and J/token, so naive is the default.

🛡️ Safety

• The core module carries #![forbid(unsafe_code)]; the crate root denies unsafe_code and only the backend modules (and the io_uring host's submission push) allow it. Every unsafe block has a SAFETY: comment; every public function is safe except the two handle constructors below.
• Handles: CuStream::from_raw and HipStream::from_raw are unsafe fn: the driver dereferences the handle, and a bogus or destroyed one faults inside the driver (CWE-822). Build them only from live streams you own.
• Use-after-free of the mapped page (CWE-416) is excluded by construction: a line accepts a store only from a stream of the context (CUDA) or device (HIP) it was made in, and its Drop synchronizes that context or device before freeing the page; if that synchronization fails, the 4 KiB page is leaked, never freed under a pending store. These rules are tested against a fake driver with no GPU (backend::mapped tests).
• The device word is never the authority. A wait returns Ok only after the stream's status query or blocking wait said done; a lost or late store costs at most one backstop interval.
• The park reads the line only through atomics and leaves the exclusive monitor armed (a clrex raises the event and turns every park into a spin).

❗ Errors

One append-only #[repr(u16)] registry, ErrorCode, with a {} form template per code; the arguments are kept and substituted only when the error is displayed. The code is the identity: a host maps it into its own registry (Error::code, Error::args) and never branches on the text.

🦀 MSRV

Rust 1.88 (edition 2024, let chains). The MSRV is raised only in a minor release.

🚧 Status

0.1.0, early release. The core, the CUDA backend and the aarch64 word wait run in production in the Metrale Engine on GB10; HIP, x86-64 word waits and Metal are compile-checked, awaiting hardware runs. Not yet published to crates.io.

⚖️ License

Licensed under either of Apache License, Version 2.0 or MIT license at your option. Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this crate by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.