Crate ferrox_core 

ferrox-core: tensor primitives, quantized matmul, RMSNorm, RoPE, and grouped-query causal attention with a simple KV cache.

CPU reference implementation. The op set and naming (RMSNorm, RoPE, GQA, KV cache) follow the now-standard vocabulary popularized by llama.cpp / vLLM / candle-transformers; the actual Rust code below is written independently. See docs/THIRD_PARTY_NOTICES.md for design credit.

The MoE expert-residency stack – expert_store (the byte budget and the SSD tier), expert_cache (which experts stay resident and the copy plans that make them so), expert_slots (the bounded slot pool behind the expert_slots::SlotDevice seam), expert_pool (the CUDA side of that seam), expert_budget (bytes in, expert slot count out), residency, placement and qstar – lives together in one crate on purpose: on unified memory two independent expert budgets are the same physical RAM counted twice. expert_store is the budget holder. The policy half is ported from FreeToken (Apache-2.0); see docs/THIRD_PARTY_NOTICES.md.

Re-exports

pub use attention::apply_rope_back;

pub use attention::apply_rope_interleaved;

pub use attention::apply_rope_interleaved_back;

pub use attention::apply_rope_interleaved_with_freq_factors;

pub use attention::apply_rope_with_freq_factors;

pub use attention::causal_gqa_attention;

pub use attention::causal_gqa_attention_paged;

pub use attention::causal_gqa_attention_paged_sinks;

pub use attention::causal_gqa_attention_prefill;

pub use attention::causal_gqa_attention_prefill_shared_kv;

pub use attention::causal_gqa_attention_prefill_shared_kv_windowed;

pub use attention::causal_gqa_attention_sinks;

pub use attention::causal_gqa_attention_softcap;

pub use attention::causal_gqa_attention_windowed;

pub use attention::causal_gqa_attention_windowed_softcap;

pub use attention::lightning_indexer_topk;

pub use cache::KvBlockPool;

pub use cache::KvCache;

pub use cache::KvPoolExhausted;

pub use cache::PagedKvCache;

pub use cache::PagedKvStore;

pub use cache::PagedStoreExhausted;

pub use cache::SharedPagedKv;

pub use csa_hca_compress::channel_gated_pool;

pub use csa_hca_compress::compress_block;

pub use deepseek_v4_attention::csa_attention;

pub use deepseek_v4_attention::hca_attention;

pub use kernel_registry::Registry as KernelRegistry;

pub use kv_block::full_blocks;

pub use kv_block::BlockHash;

pub use kv_block::BlockHasher;

pub use kv_disk::decode_block;

pub use kv_disk::encode_block;

pub use kv_disk::encoded_len;

pub use kv_disk::BlockFormatError;

pub use kv_disk::DiskConfig;

pub use kv_disk::DiskKvStore;

pub use kv_disk::DiskStats;

pub use kv_disk::ReadHandle;

pub use kv_disk::ReadOutcome;

pub use kv_disk::StoreError;

pub use kv_signature::CacheSignature;

pub use kv_signature::KvBlock;

pub use kv_signature::KvDtype;

pub use kv_signature::SignatureError;

pub use kv_signature::UnverifiedBlock;

pub use kv_signature::BLOCK_FORMAT_VERSION;

pub use kv_signature::READABLE_FORMAT_VERSIONS;

pub use kv_swa::aligned_block_size;

pub use kv_swa::BlockLayout;

pub use kv_swa::BlockLayoutError;

pub use matmul::geglu;

pub use matmul::gelu;

pub use matmul::matmul_f32;

pub use matmul::rms_norm;

pub use matmul::rms_norm_per_head;

pub use matmul::silu;

pub use matmul::situ_and_mul;

pub use matmul::softcap_inplace;

pub use matmul::swiglu;

pub use tensor::Tensor;

pub use weight_matrix::active_backend;

pub use weight_matrix::cpu_int_dot_kind_supported;

pub use weight_matrix::cuda_matvec_kind_supported;

pub use weight_matrix::metal_matvec_kind_name;

pub use weight_matrix::metal_mul_mm_kind_supported;

pub use weight_matrix::BatchActs;

pub use weight_matrix::QuantKind;

pub use weight_matrix::WeightMatrix;

Modules

attention

Rotary position embedding (RoPE, both the split-half apply_rope and interleaved apply_rope_interleaved conventions) and grouped-query causal attention (GQA). This is the “vanilla” attention path used as the correctness baseline. causal_mla_attention/causal_mla_attention_sparse add DeepSeek-style latent attention and its DSA sparse-selection variant (GLM-5.2, DeepSeek V3.2/V4); both mechanisms are now backed by real, public reference implementations (see docs/MODELS.md). ferrox_models::mla/ferrox_models::glm_dsa compose these primitives into full RoPE-carrying MLA forward passes.

bench_profile

Where a machine’s measured bandwidth profile lives, and when it may be trusted.

block_sparse

MiniMax-M3’s block-sparse attention selection: which 128-token KV blocks a query may look at.

cache

A per-layer KV cache, growable one position at a time during decode. Two growth strategies exist:

csa_hca_compress

DeepSeek V4’s CSA (Compressed Sparse Attention) / HCA (Heavily Compressed Attention) shared block-compression primitive: pooling a block of raw per-token key/value-role vectors down to one compressed entry, before RMSNorm and RoPE-on-the-rope-slice.

deepseek_v4_attention

DeepSeek V4’s CSA (Compressed Sparse Attention) / HCA (Heavily Compressed Attention) attention assembly: combining a small raw local window with a set of already-compressed entries (see crate::csa_hca_compress for how those entries are produced).

expert_budget

Splitting a memory budget between the GPU expert cache and the KV pool.

expert_cache

The global expert slot cache: which experts are resident on the GPU right now, and what one decode step has to move.

expert_pool

Device-side homes for the expert slot pool crate::expert_slots::ExpertSlots governs.

expert_slots

The executor for crate::expert_cache’s plans: a bounded slot pool, the copies a plan asks for, and the counter that says a warm step moved nothing.

expert_store

A bounded, lease-protected byte cache for routed-expert weights – the storage foundation for running MoE checkpoints whose experts do not all fit in RAM at once (stream cold experts from SSD, keep hot ones resident under one global byte budget).

host_memory

How much memory the host actually has free, and whether a model fits.

instance

Who else is already running a model on this box.

kernel_registry

Sealed kernel-lookup registry: makes a missing kernel loud instead of silently slow.

kv_block

Content-addressed identity for KV-cache blocks: what makes two stored prefixes the same prefix, across processes and across restarts.

kv_disk

The disk tier for the KV prefix cache: where a block goes so that a prefix survives eviction from RAM, and a process restart.

kv_signature

Compatibility marking for stored KV blocks: whether a block found in a cache may be used, as opposed to merely found.

kv_swa

Block-size alignment for sliding-window attention (SWA).

matmul

placement

Which MoE layers decode on the CPU.

qstar

The q* policy: bandwidth-adaptive CPU/GPU expert execution.

residency

What a layer’s expert bank actually settled at in host memory, and how much of host memory this machine will let us page-lock at all.

summary_stats

Summary statistics over samples that may be missing.

tensor

A minimal row-major dense f32 tensor. Ferrox keeps this deliberately small: weights live quantized in the mmap’d GGUF file and are dequantized on demand by ferrox-quant; this type is for activations and small dequantized weight slices during the forward pass.

threads

CPU worker-pool policy, shared by ferrox (CLI) and ferrox-server.

turboquant

Walsh–Hadamard transform helpers for TurboQuant-style KV compression.

vexp

The ggml_v_expf polynomial, shared.

weight_matrix

WeightMatrix: a weight matrix that may live either as plain f32 (small dims, embeddings, synthetic test weights) or as raw Q8_0/Q4_0 block bytes loaded straight from a GGUF file, with no f32 expansion at load time. This is what lets ferrox load a multi-billion-parameter checkpoint without first blowing it up 4x in RAM: the loader (ferrox-models) hands tensors over still quantized, and every matmul call here dispatches to the fused dequant+dot kernels in ferrox-quant.