Module cache 

A per-layer KV cache, growable one position at a time during decode. Two growth strategies exist:

• with_pool: PagedAttention-style block allocation. Many caches (one per concurrent request, typically) draw fixed-size blocks from one shared, bounded KvBlockPool instead of each independently pre-committing to a worst-case context length. Growth happens in fixed block-sized quanta, and a cache’s blocks return to the shared pool when it’s dropped, so the pool’s free count is a real, live admission-control signal a caller can check before accepting a new request. This is the block-allocation half of PagedAttention; it does not (yet) change how attention reads a cache – k/v are still read as one contiguous slice per sequence (see Decoder::forward_token/forward_batch), just backed by capacity that grows in block-sized steps instead of Rust’s default exponential Vec growth. Wiring this into ferrox-server as live per-request admission control via FERROX_KV_POOL_BLOCKS/FERROX_KV_POOL_BLOCK_SIZE.

Structs

KvBlockPool

A bounded pool of fixed-size KV-cache blocks (in positions) shared across many KvCache instances, typically one pool per server process. Each KvCache::with_pool acquires one block up front and one more each time it grows past its currently held capacity; free_blocks is therefore a live, accurate admission-control signal – a caller can check it before accepting a new request rather than discovering exhaustion only after committing memory.

KvCache

KvPoolExhausted

Returned by KvCache::push (and with_pool) when a pool-backed cache needs another block but its shared KvBlockPool has none free. Caches built with new/with_capacity never return this – their growth is unconditional, matching their pre-paging behavior exactly.

PageGroup

A handle to one block in every layer.

PagedKvCache

One sequence’s view into a shared PagedKvStore: a block table (which physical blocks this sequence’s positions live in, in order) plus how many positions have been written so far. Unlike KvCache, this holds no K/V data itself – every read and write goes through the shared store.

PagedKvStore

The other half of PagedAttention that KvBlockPool/KvCache::with_pool deliberately don’t implement (see this module’s doc comment): real, shared physical block storage that many sequences’ block tables can address into, instead of each KvCache still owning its own private, contiguous Vec. KvBlockPool only ever bounds a count of blocks each cache may grow to; PagedKvStore is the actual backing memory, and a sequence’s PagedKvCache holds a block table (an ordered list of block IDs into this shared store) instead of owning K/V data directly. This is what makes non-contiguous-block reads during attention (causal_gqa_attention_paged, in attention.rs) possible at all – causal_gqa_attention’s existing contiguous-slice read pattern has no way to express “position 37 lives in block 12, cached out of order relative to block 5.”

PagedStoreExhausted

Returned when a PagedKvCache needs another block but its PagedKvStore has none free – the paged-storage analog of KvPoolExhausted.

SharedPagedKv

Per-layer PagedKvStores that many concurrent requests share.