Enum HostBuf 

pub enum HostBuf {
    Paged(Vec<u8>),
    Pinned {
        slice: Arc<PinnedHostSlice<u8>>,
        base: *const u8,
        len: usize,
    },
    PinnedAlias {
        owner: Arc<HostBuf>,
        base: *const u8,
        len: usize,
    },
    Mmap {
        map: Arc<Mmap>,
        file: Arc<File>,
        off: usize,
        len: usize,
    },
}

One layer’s stacked 256-expert tensor, raw GGUF quant bytes held HOST-RESIDENT.

EDGE-1: these bytes are NEVER uploaded at load (uploading 29.75GB would OOM a 24GB GPU — this is BUG-4). Per token, only the 8 routed experts are staged H2D into a small GPU scratch.

ne = [in_f, out_f, n_expert]; the expert axis (ne[2]) is the slowest/highest-stride axis, so expert e occupies the CONTIGUOUS byte block bytes[e*expert_stride .. (e+1)*expert_stride].

THE 3D FIX: GpuTensor::load computes row_bytes = raw.len()/ne[1], which for a stacked 3D tensor ignores the 256-expert axis and is 256x too large (gate_exps -> 430080 instead of 1680). load() here uses row_bytes = raw.len() / (out_f * n_expert) (= 1680 gate/up, 544 down). Host byte storage for the expert blocks. Default = a pageable Vec<u8> (current behavior). Under MEMRA_MOE_PINNED (auto-on when MEMRA_MOE_CACHE is set), the bytes live in CUDA pinned host memory so the miss-path memcpy_htod is a true DMA, not a pageable bounce copy (MOE-SLRU-PLAN §C.1).

CAVEAT (§C.1): alloc_pinned uses CU_MEMHOSTALLOC_WRITECOMBINED — great for H2D-only (the expert bytes are never read by the CPU on the hot path), but write-combined memory is SLOW for CPU reads. A future CPU-VNNI cold-expert fallback must NOT read from this buffer.

Variants

Paged(Vec<u8>)

Pinned

Pinned host memory. We keep the PinnedHostSlice alive (it owns the allocation; Drop frees it) AND cache its raw base pointer + len so the hot-path as_bytes() needs no per-call event sync.

Fields

slice: Arc<PinnedHostSlice<u8>>

base: *const u8

len: usize

PinnedAlias

Alias into a shared pinned slab (ST pinned tier): owner keeps the slab alive; base/len select this expert’s window. Same DMA class as Pinned.

Fields

owner: Arc<HostBuf>

base: *const u8

len: usize

Mmap

SPILLING-PLAN §1, Tier 2 (disk): the bytes live in an mmap’d region of the GGUF file, NOT in RAM. map is MAP_SHARED, no MAP_POPULATE — zero upfront copy. The first memcpy_htod of this slice page-faults → NVMe read → DMA (the demand-fault disk path). off/len select this expert’s contiguous block within the shared file mmap. Bit-identical to Paged/Pinned — those copied FROM exactly these on-disk bytes, so the GEMM result is unchanged.

Fields

map: Arc<Mmap>

file: Arc<File>

The same opened inode backing map. It must outlive the loader source so future explicit positioned reads cannot accidentally reopen a replaced path.

off: usize

Absolute byte offset within both the whole-file mmap and file.

len: usize

Implementations

impl HostBuf

pub fn as_bytes(&self) -> &[u8]

pub fn len(&self) -> usize

pub fn advise_willneed(&self, rel_off: usize, len: usize) -> bool

Best-effort OS read-ahead for a future mmap-backed expert range. This does not touch or copy the bytes, so the zero-copy ownership contract is unchanged. Non-mmap buffers are already resident and need no advice. Kept fallible-at-the-OS but non-fatal at the call site: an unsupported/pressured kernel simply leaves the normal demand-fault path in place.

Trait Implementations

impl Send for HostBuf

impl Sync for HostBuf