Crate ferrox_core 

ferrox-core: tensor primitives, quantized matmul, RMSNorm, RoPE, and grouped-query causal attention with a simple KV cache.

CPU reference implementation. The op set and naming (RMSNorm, RoPE, GQA, KV cache) follow the now-standard vocabulary popularized by llama.cpp / vLLM / candle-transformers; the actual Rust code below is written independently. See docs/THIRD_PARTY_NOTICES.md for design credit.

Re-exports

pub use attention::apply_rope_back;

pub use attention::apply_rope_interleaved;

pub use attention::apply_rope_interleaved_back;

pub use attention::apply_rope_interleaved_with_freq_factors;

pub use attention::apply_rope_with_freq_factors;

pub use attention::causal_gqa_attention;

pub use attention::causal_gqa_attention_paged;

pub use attention::causal_gqa_attention_prefill;

pub use attention::causal_gqa_attention_prefill_shared_kv;

pub use attention::causal_gqa_attention_prefill_shared_kv_windowed;

pub use attention::causal_gqa_attention_sinks;

pub use attention::causal_gqa_attention_softcap;

pub use attention::causal_gqa_attention_windowed;

pub use attention::causal_gqa_attention_windowed_softcap;

pub use attention::lightning_indexer_topk;

pub use cache::KvBlockPool;

pub use cache::KvCache;

pub use cache::KvPoolExhausted;

pub use cache::PagedKvCache;

pub use cache::PagedKvStore;

pub use cache::PagedStoreExhausted;

pub use csa_hca_compress::channel_gated_pool;

pub use csa_hca_compress::compress_block;

pub use deepseek_v4_attention::csa_attention;

pub use deepseek_v4_attention::hca_attention;

pub use kernel_registry::Registry as KernelRegistry;

pub use kv_block::full_blocks;

pub use kv_block::BlockHash;

pub use kv_block::BlockHasher;

pub use kv_disk::decode_block;

pub use kv_disk::encode_block;

pub use kv_disk::encoded_len;

pub use kv_disk::BlockFormatError;

pub use kv_disk::DiskConfig;

pub use kv_disk::DiskKvStore;

pub use kv_disk::DiskStats;

pub use kv_disk::ReadHandle;

pub use kv_disk::ReadOutcome;

pub use kv_disk::StoreError;

pub use kv_signature::CacheSignature;

pub use kv_signature::KvBlock;

pub use kv_signature::KvDtype;

pub use kv_signature::SignatureError;

pub use kv_signature::UnverifiedBlock;

pub use kv_signature::BLOCK_FORMAT_VERSION;

pub use kv_signature::READABLE_FORMAT_VERSIONS;

pub use kv_swa::aligned_block_size;

pub use kv_swa::BlockLayout;

pub use kv_swa::BlockLayoutError;

pub use matmul::geglu;

pub use matmul::gelu;

pub use matmul::matmul_f32;

pub use matmul::rms_norm;

pub use matmul::rms_norm_per_head;

pub use matmul::silu;

pub use matmul::situ_and_mul;

pub use matmul::softcap_inplace;

pub use matmul::swiglu;

pub use tensor::Tensor;

pub use weight_matrix::active_backend;

pub use weight_matrix::cpu_int_dot_kind_supported;

pub use weight_matrix::cuda_matvec_kind_supported;

pub use weight_matrix::metal_matvec_kind_name;

pub use weight_matrix::metal_mul_mm_kind_supported;

pub use weight_matrix::BatchActs;

pub use weight_matrix::QuantKind;

pub use weight_matrix::WeightMatrix;

Modules

attention

Rotary position embedding (RoPE, both the split-half apply_rope and interleaved apply_rope_interleaved conventions) and grouped-query causal attention (GQA). This is the “vanilla” attention path used as the correctness baseline. causal_mla_attention/causal_mla_attention_sparse add DeepSeek-style latent attention and its DSA sparse-selection variant (GLM-5.2, DeepSeek V3.2/V4); both mechanisms are now backed by real, public reference implementations (see docs/MODELS.md). ferrox_models::mla/ferrox_models::glm_dsa compose these primitives into full RoPE-carrying MLA forward passes.

cache

A per-layer KV cache, growable one position at a time during decode. Two growth strategies exist:

csa_hca_compress

DeepSeek V4’s CSA (Compressed Sparse Attention) / HCA (Heavily Compressed Attention) shared block-compression primitive: pooling a block of raw per-token key/value-role vectors down to one compressed entry, before RMSNorm and RoPE-on-the-rope-slice.

deepseek_v4_attention

DeepSeek V4’s CSA (Compressed Sparse Attention) / HCA (Heavily Compressed Attention) attention assembly: combining a small raw local window with a set of already-compressed entries (see crate::csa_hca_compress for how those entries are produced).

expert_store

A bounded, lease-protected byte cache for routed-expert weights – the storage foundation for running MoE checkpoints whose experts do not all fit in RAM at once (stream cold experts from SSD, keep hot ones resident under one global byte budget).

instance

Who else is already running a model on this box.

kernel_registry

Sealed kernel-lookup registry: makes a missing kernel loud instead of silently slow.

kv_block

Content-addressed identity for KV-cache blocks: what makes two stored prefixes the same prefix, across processes and across restarts.

kv_disk

The disk tier for the KV prefix cache: where a block goes so that a prefix survives eviction from RAM, and a process restart.

kv_signature

Compatibility marking for stored KV blocks: whether a block found in a cache may be used, as opposed to merely found.

kv_swa

Block-size alignment for sliding-window attention (SWA).

matmul

tensor

A minimal row-major dense f32 tensor. Ferrox keeps this deliberately small: weights live quantized in the mmap’d GGUF file and are dequantized on demand by ferrox-quant; this type is for activations and small dequantized weight slices during the forward pass.

threads

CPU worker-pool policy, shared by ferrox (CLI) and ferrox-server.

turboquant

Walsh–Hadamard transform helpers for TurboQuant-style KV compression.

weight_matrix

WeightMatrix: a weight matrix that may live either as plain f32 (small dims, embeddings, synthetic test weights) or as raw Q8_0/Q4_0 block bytes loaded straight from a GGUF file, with no f32 expansion at load time. This is what lets ferrox load a multi-billion-parameter checkpoint without first blowing it up 4x in RAM: the loader (ferrox-models) hands tensors over still quantized, and every matmul call here dispatches to the fused dequant+dot kernels in ferrox-quant.