Crate memra_runtime 

memra inference runtime. Correctness-first: every GPU op is validated against a CPU reference before any sm_120 fast-path replaces it.

Re-exports

pub use memra_gguf;

Structs

Gpu

GPU runtime handle: a context + stream + cuBLASLt.

GpuMainOverride

A rank-local CUDA scope nested inside another engine’s PP stage scope.

Rank0RedirectGuard

RAII scope for the rank0 redirect: clears on drop (panic-safe).

StreamOverride

RAII scope: while alive, Gpu::stream() on THIS thread returns the pushed stream. Nest freely (stack). Popping on Drop keeps panic paths consistent.

Functions

cpu_linear

CPU reference matmul for a linear layer y = x @ W^T. Conventions (ggml/GGUF): a weight tensor with ne=[in, out] is stored row-major as out rows of in contiguous elements — i.e. W[oin + i]. A linear layer computes y[o] = sum_i x[i] * W[oin + i], for each of out outputs. Batched over m tokens: x: [m, in] row-major (x[tin + i]); w: [out, in] row-major (w[oin + i]); y: [m, out].

decode_phase

push_stream_override

Push a matched stream/cuBLASLt binding for the current thread until the guard drops.

rank0_redirect_scope

set_decode_phase

TOKEN-PIPELINE phase (step37 chain): while Some(p), enter_main binds each gpu’s phase-p stream instead of its main stream, so alternate tokens’ rank-local work rides disjoint streams. Cross-stream ordering is the SETTER’s job (the multi-stream law above): the chain wires per-layer KV events between phases.

set_rank0_redirect

Install/clear the rank0 redirect (ctx ordinal + stream/blas of the model engine).