Module gpu_mem_autosize 

GPU memory auto-tuning for the KV pool.

Reads model config + on-disk weight file sizes + nvidia-smi reported GPU total, then sets FERRUM_KV_MAX_BLOCKS so the KV pool fits inside total_mem * gpu_memory_utilization after weights and a scratch reserve. Mirrors vLLM’s gpu_memory_utilization knob (default 0.9).

Skipped when:

• nvidia-smi missing (Mac / CPU-only): keep static defaults.
• config.json not parseable: keep static defaults.
• User explicitly set FERRUM_KV_MAX_BLOCKS: respect their override.

Structs

AutoSizeResult

Enums

AutoSizeProfile

CLI usage profile: which presets the autosizer should consider.

Functions

apply_auto_size

Apply auto-sizing: read CLI flag, query nvidia-smi, set env vars. Sets FERRUM_KV_MAX_BLOCKS (global physical paged-KV block budget), FERRUM_PAGED_MAX_SEQS (scheduler/model concurrency shape), and FERRUM_KV_CAPACITY (per-sequence logical table stride). The model allocates the GPU KV pool from KV_MAX_BLOCKS; PAGED_MAX_SEQS * KV_CAPACITY no longer reserves physical KV blocks up front.

apply_auto_size_with_profile

Apply auto-sizing with explicit usage profile. The chat profile flips priority — long context per seq beats wide batch — because the CLI REPL only ever has one active sequence and multi-turn dialogues blow past the default 512-token cap fast.

auto_size_kv_blocks

Compute target FERRUM_KV_MAX_BLOCKS from gpu_memory_utilization.

auto_size_kv_blocks_with_pool_copies