gpu-async-reactor

Await GPU work as a memory descriptor. A GPU stream stores a sequence number into a word of host-visible memory behind its work. That word is to device completion what a file descriptor is to I/O readiness: this crate makes it a first-class awaitable source, and supplies the wait plans, the deadline estimator and a small executor around it, so a host thread can sleep until the GPU is nearly done, park in hardware on the completion word, and run other ready work in between.

📑 Contents

• 🚀 Quick start
• 🧭 Design
• 🔌 Backends and hosts
• ⚙️ Choosing an executor
• 📊 Benchmarks
• 🛡️ Safety
• ❗ Errors
• 🦀 MSRV
• 🚧 Status
• ⚖️ License

🚀 Quick start

[dependencies]
gpu-async-reactor = { version = "0.1", features = ["cuda"] }

Wait for a CUDA stream with the plan the Metrale serve A/B picked (park on the completion word, query the stream every millisecond as the error authority):

use gpu_async_reactor::backend::cuda::{CuStream, CudaPort, MappedLine};
use gpu_async_reactor::backend::wordwait::WordWait;
use gpu_async_reactor::host::std_host::StdClock;
use gpu_async_reactor::{Slot, WaitPlan, run_inline, wait};

let plan = WaitPlan::parse("flag:park_us=100,backstop_us=1000")?;
let line = MappedLine::new()?;                       // one 64-byte line, device-mapped, owned by
                                                     // (and retaining) the device's primary context
let stream = unsafe { CuStream::from_raw(raw) };     // your live CUstream, on the primary context
let (clock, mut seq) = (StdClock::new(), 0);
// ... enqueue work on `stream` ...
let mut port = CudaPort::new(&line, stream, Slot::new(0).unwrap(), &mut seq, WordWait::probe().ok());
let report = run_inline(wait(&plan, None, &mut port, &clock, &clock), "my wait")??;
println!("{:?} after {} ns, {} parks", report.kind, report.wait_ns, report.parks);

Run several waits and CPU work on one thread (no GPU needed for this one):

cargo run --example fill_executor --features std,wordwait
cargo run --release --example cuda_wait --features cuda      # needs an NVIDIA GPU

🧭 Design

One crate, organised as ports and hosts (SBIO): the core holds every decision and does no I/O; backends and hosts are generic parameters chosen once by the binary. There is no dyn in the core.

The wait, step by step:

block?  ── yes ──▶ port.block()                                   ▶ Blocked
  │ no
query() ── done ─▶                                                 ▶ Immediate (exact residual)
  │ pending
flag? ── arm(): enqueue store(seq) behind the work
predict? ── Timer.sleep_until(start + predicted − margin); query() ── done ─▶ Late (censored)
park: flag ── park(park_us) until the word reaches seq, query() every backstop_us
      poll ── query() until done                                   ▶ Parked (exact residual)

🔌 Backends

Every vendor entry point is resolved by name at run time (libcuda.so.1, libamdhip64.so), so the crate links nothing and a machine without the driver fails only the constructor. Adding hardware means one LineDriver table (the CUDA and HIP lines share the ownership logic) or one CompletionPort.

⚙️ Choosing an executor

resolve(requested, strictness, &caps) walks the chain once at startup. With Strictness::Fallback an unavailable executor yields the fallback plus a coded ExecutorFallback warning naming what was asked, why it is unavailable, and what runs; with Strictness::Strict it is an ExecutorRefused error. Declared is the generic shape: implement it for any other setting with a fallback chain.

📊 Benchmarks

All numbers are from the Metrale Engine on an NVIDIA GB10 (Grace Arm + Blackwell, coherent memory), CUDA 13, serving through the in-tree predecessor of this crate (identical logic). Method: same-session, rotated boots per arm, clean box (load < 4, no other GPU work), ratios paired to the spin baseline booted in the same rotation, 95 % CIs (Student-t on paired ratios); GPU energy from the NVML total-energy counter read twice per cell.

Wake latency and park cost (micro):

Serve A/B, Qwen3.6-35B-A3B FP8, n = 5 rotated boots, ratio to the spinning cuStreamSynchronize:

Reading: parking on the word holds throughput parity with a spin; adding the predicted sleep gives back two thirds of the scheduler core at C4-C16 for a 0.2-0.5 % throughput cost, and is at parity from C32. Waking from an interrupt or a file descriptor instead lost 0-3 % (35B) and up to 9 % (27B, C8) and multiplied context switches 9-50x per token. CPU power of a WFE park is not on any GB10 rail; only the GPU rail is metered. The executor A/B (naive vs gpu-reactor vs io-uring, same method, C1-C64) measured them equal within noise in tok/s, CPU and J/token, so naive is the default.

🛡️ Safety

• The core module carries #![forbid(unsafe_code)]; the crate root denies unsafe_code and only the backend modules (and the io_uring host's submission push) allow it. Every unsafe block has a SAFETY: comment; every public function is safe except the two handle constructors below.
• Handles: CuStream::from_raw and HipStream::from_raw are unsafe fn: the driver dereferences the handle, and a bogus or destroyed one faults inside the driver (CWE-822). Build them only from live streams you own.
• The mapped page outlives every other owner of its context. A line belongs to the primary context of the current context's device and holds its own reference to it (cuDevicePrimaryCtxRetain; HIP: hipDevicePrimaryCtxRetain) from before the page is allocated until after it is freed. Another owner releasing the context, or destroying a non-primary context that was current when the line was made, cannot free the page. A store is accepted only from a stream of that primary context (CUDA) or device (HIP): a stream of a cuCtxCreate context, which the line cannot keep alive, is refused with ForeignStream. Drop synchronizes the context before freeing the page, then releases it; if that synchronization fails, the page and the reference are leaked, never freed under a pending store. The ordering is tested against a fake driver with no GPU (backend::mapped tests) and on a GB10 (tests/cuda_line_lifetime.rs, #[ignore]d without a GPU).
  • Fixed in 0.1.1 (CWE-416; advisory pending): 0.1.0 claimed this was excluded by construction, and it was not: the line took no reference to its context, so safe code that dropped its context owner before the line (for example cudarc's CudaContext) left the line reading freed memory (SIGSEGV), and a new line could be handed the same page. Upgrade to 0.1.1.
  • What remains assumed: nothing resets the device while a line lives (cuDevicePrimaryCtxReset, cudaDeviceReset, hipDeviceReset). A reset frees every allocation of every user in the process regardless of references, so a wrapper that exposes it as safe is unsound on its own. cuCtxDestroy on a retained primary context is refused by the driver (measured: status 201, the line stays valid).
• A word-wait capability cannot be forged. WordWait has private fields; its only constructor is WordWait::probe, which reads the CPU (on x86-64, only CPUID leaves the CPU reports). 0.1.0 had public fields, so safe code could build a WordWait claiming an instruction the core lacks, and park would execute it (SIGILL, CWE-754).
• The device word is never the authority. A wait returns Ok only after the stream's status query or blocking wait said done; a lost or late store costs at most one backstop interval.
• The park reads the line only through atomics and leaves the exclusive monitor armed (a clrex raises the event and turns every park into a spin).

❗ Errors

One append-only #[repr(u16)] registry, ErrorCode, with a {} form template per code; the arguments are kept and substituted only when the error is displayed. The code is the identity: a host maps it into its own registry (Error::code, Error::args) and never branches on the text.

🦀 MSRV

Rust 1.88 (edition 2024, let chains). The MSRV is raised only in a minor release.

🚧 Status

0.1.1, early release. The core, the CUDA backend and the aarch64 word wait run in production in the Metrale Engine on GB10; HIP, x86-64 word waits and Metal are compile-checked, awaiting hardware runs.

⚖️ License

Licensed under either of Apache License, Version 2.0 or MIT license at your option. Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this crate by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.