Skip to main content

Module chunked_prefill

Module chunked_prefill 

Source
Expand description

§Chunked Prefill Scheduler

Provides chunked prefill scheduling to prevent head-of-line blocking and memory spikes during long context ingestion.

Large prompts (e.g. 2k–8k tokens) are segmented into configurable chunks (e.g. 128–512 tokens), interleaving prompt KV-cache prefilling with active speculative decoding bursts.

Structs§

ChunkedPrefillConfig
Configuration for chunked prefill processing.
ChunkedPrefillScheduler
Chunked prefill orchestrator.
PrefillTask
An active prompt ingestion task being processed in chunks.