Expand description
§Chunked Prefill Scheduler
Provides chunked prefill scheduling to prevent head-of-line blocking and memory spikes during long context ingestion.
Large prompts (e.g. 2k–8k tokens) are segmented into configurable chunks (e.g. 128–512 tokens), interleaving prompt KV-cache prefilling with active speculative decoding bursts.
Structs§
- Chunked
Prefill Config - Configuration for chunked prefill processing.
- Chunked
Prefill Scheduler - Chunked prefill orchestrator.
- Prefill
Task - An active prompt ingestion task being processed in chunks.