Modules§
- approx
- Approximate KV Indexer
- indexer
- KV RadixTree
- metrics_
aggregator - protocols
- publisher
- recorder
- scheduler
- scoring
- Scoring functions for the KV router.
- sequence
- KV Cache Sequence Management for LLM Inference
- subscriber
- Background processes for the KV Router including event consumption and snapshot uploads.
Structs§
- KvPush
Router - KvRouter
- A KvRouter only decides which worker you should use. It doesn’t send you there. TODO: Rename this to indicate it only selects a worker, it does not route.
- KvRouter
Config - KV Router configuration parameters
- Router
Config Override - Override configuration for router settings that can be specified per-request
- Router
Config Override Builder - Builder for
RouterConfigOverride.
Enums§
- Indexer
- Router
Config Override Builder Error - Error type for RouterConfigOverrideBuilder
Constants§
- ACTIVE_
SEQUENCES_ SUBJECT - KV_
EVENT_ SUBJECT - KV_
HIT_ RATE_ SUBJECT - KV_
METRICS_ ENDPOINT - KV_
METRICS_ SUBJECT - PREFILL_
SUBJECT - RADIX_
STATE_ BUCKET - RADIX_
STATE_ FILE - ROUTER_
CLEANUP_ LOCK - ROUTER_
SNAPSHOT_ LOCK
Traits§
- Worker
Selector - A trait that users can implement to define custom selection logic