Expand description
§onnx-runtime-ep-cpu
The CPU execution provider for the ORT 2.0 runtime (see docs/ORT2.md §4.4
and §54 Phase 1). It implements onnx_runtime_ep_api::ExecutionProvider
and hosts pure-Rust reference kernels for the Phase-1 op set (MatMul,
Add, Relu, Reshape, Transpose, Gather, LayerNormalization).
§Backends: correctness baseline + SIMD fast path
The GEMM hot spot is served through backend::CpuBackend (docs/ORT2.md
§25.2). The default backend is a pure-Rust blocked, register-tiled,
rayon-parallelized f32 GEMM — the portable, offline correctness baseline that
compiles anywhere with no C++/FFI. On supported x86 hosts, the built-in
SimdX86 implementation provides the default fast path. Every backend lives behind the
onnx_runtime_ep_api::Kernel trait, so neither the EP contract nor the
session observes which one ran. See kernels::matmul for the hot spot.
§unsafe
The default (Generic) path is unsafe-minimal: the only unsafe is the raw
device-buffer access the ep-api contract forces (aligned host
alloc/dealloc, memcpy, and strided element reads/writes), each isolated
and SAFETY-documented. The blocked rayon GEMM itself contains no unsafe;
all kernel arithmetic is safe Rust operating on
dense Vec<f32> buffers produced by the two audited accessors in kernels.
Re-exports§
pub use backend::CpuBackend;pub use kernels::qmoe::WeightOffloadHostCache;pub use optimizer::ConvBatchNormActivationFusion;pub use optimizer::MatMulNBitsBiasFusion;pub use optimizer::ProjectionFusion;pub use optimizer::cpu_optimization_passes;pub use provider::CpuExecutionProvider;pub use weight_offload::placement::ArbitrationAction;pub use weight_offload::placement::GpuLayersOverrideReport;pub use weight_offload::placement::HostFallbackReason;pub use weight_offload::placement::IqFormat;pub use weight_offload::placement::KvAdmissionDecision;pub use weight_offload::placement::KvAdmissionLimitingFactor;pub use weight_offload::placement::LayerPlacement;pub use weight_offload::placement::LayerWeightRegions;pub use weight_offload::placement::Placement;pub use weight_offload::placement::PlacementError;pub use weight_offload::placement::PlacementPlan;pub use weight_offload::placement::QuantTileFormat;pub use weight_offload::placement::RegionPlacement;pub use weight_offload::placement::SnappedTileSize;pub use weight_offload::placement::TileSizeError;pub use weight_offload::placement::VramArbitrationConfig;pub use weight_offload::placement::VramArbitrationError;pub use weight_offload::placement::VramArbitrationOutcome;pub use weight_offload::placement::VramArbitrationState;pub use weight_offload::placement::VramDemand;pub use weight_offload::placement::VramSubBudgets;pub use weight_offload::placement::arbitrate_vram;pub use weight_offload::placement::decide_kv_admission;pub use weight_offload::placement::plan_placement;pub use weight_offload::placement::snap_transfer_tile_bytes;pub use weight_offload::LinuxProcessMemoryStats;pub use weight_offload::WEIGHT_OFFLOAD_ENV;pub use weight_offload::WEIGHT_OFFLOAD_HOST_BYTES_ENV;pub use weight_offload::WeightOffloadLayerStats;pub use weight_offload::WeightOffloadStats;pub use weight_offload::set_weight_offload_host_budget;pub use weight_offload::weight_offload_stats;pub use kernels::selection::non_max_suppression;pub use kernels::slice::SliceAxisPlan;pub use kernels::slice::slice_axes_steps;pub use kernels::slice::slice_plan;pub use kernels::matmul_nbits::set_decode_thread_budget;pub use kernels::matmul_nbits::bound_process_to_decode_budget;pub use kernels::matmul_nbits::with_decode_pool_scope;
Modules§
- backend
- CPU GEMM backend selection (
docs/ORT2.md§25.2 “CPU Backend Strategy”). - decode_
affinity - NUMA-aware CPU affinity for the bounded M=1 decode thread pool.
- decode_
numa - Two-level NUMA-split decode layout for the bounded M=1 decode pool.
- decode_
spmd - Persistent SPMD decode pool: one hot worker set joined by a lightweight reusable barrier, replacing the ~141 per-token Rayon fork-join regions.
- dtype
- Reusable dtype-generic machinery for the arithmetic CPU kernels
(
docs/ORT2.md§4.4; project preference “不同的dtype，是不是可以用 template”). - kernels
- CPU kernels for the Phase-1 BERT-on-CPU correctness milestone (
docs/ORT2.md§4.4). One [Kernel] per ONNX op, keyed purely by op type — there are no model-specific shapes or names anywhere in this crate; BERT is only the validation target. - optimizer
- provider
- The
CpuExecutionProvider: a host execution provider backed by pure-Rust reference kernels (docs/ORT2.md§4.4). - strided
- Strided-view access helpers shared by the CPU kernels (
docs/ORT2.md§5). - weight_
offload - Huge-model weight-offload mode and lightweight process-wide observability.
Macros§
- dispatch_
arith - Map a runtime
DataTypeto a monomorphized body over the matching Rust element type, across the full ONNX numeric set (floats + signed/unsigned integers). Binds$Tvia a localtypealias; unsupported dtypes yield a RULE #1 error. The body must evaluate toResult<()>. - dispatch_
float - Like [
dispatch_arith] but restricted to the floating-point dtypes ONNX defines transcendental / accumulate ops over (f32,f16,bf16,f64).
Structs§
- Execution
Provider Capabilities - Lazy
Weight - Phase3a
Host Only Binder - CPU-only Phase-3a binder: callers must use the host materialization route.
- Resident
Weight
Enums§
- Lazy
Weight Boundary - Negotiated
Weight - Weight
Handle - General executor weight input: resident today, lazy when an EP opts in.
- Weight
Handle Error
Constants§
- NXRT_
WEIGHT_ PAGING_ CAPABILITY - Capability flag advertised by paging-aware execution providers.
Traits§
- Lazy
Device Weight Binder - EP seam for Phase-3b live device paging.
- Resident
Weight Materializer