Module expert_budget 

Splitting a memory budget between the GPU expert cache and the KV pool.

Why the expert cache gets first claim

Both pools compete for the same bytes, but they do not degrade the same way. A KV pool one page short means one fewer concurrent request or a shorter context – a scheduling limit, felt as a queue. An expert cache one slot short means every step that routes to the missing expert pays a PCIe transfer or a CPU detour – a per-token tax on every request, forever. So plan_cache_budget fills the expert cache first, up to full residency, and gives the remainder to KV – with a floor of kv_reserve_pages, because a server that cannot hold a context serves nothing at all.

A byte budget becomes a slot count

A user says a number of bytes; a bounded pool of fixed-size slots is the only thing you can actually cap. expert_bytes_per_slot prices one slot as the sum of its row across every weight bank, and plan_cache_budget divides. It lives beside expert_store, which holds the budget it sizes, and beside expert_cache, whose slots it counts.

Ported 1:1 from FreeToken’s engine/cache_budget.py (Apache-2.0); see docs/THIRD_PARTY_NOTICES.md.

Structs

BudgetTooSmall

A split that does not fit, refused before anything was freed.

PoolSizes

A pool split.

Functions

expert_bytes_per_slot

What one expert costs in the GPU cache: the sum of its row across every weight bank.

net_cache_budget_bytes

The VRAM a rebuild may spend: a fraction of what was free before the weights loaded, minus the weights, minus whatever the pools need unconditionally.

plan_cache_budget

Split budget_bytes between the expert cache and the KV pool.

required_bytes

What a given split would actually cost.

startup_kv_budget

The KV budget at startup, when the weights have just been loaded.