Expand description
Runtime-neutral mock inference schedulers and attention-DP composition.
Modules§
- generalized
- Runtime-neutral mock-engine composition.
Structs§
- Admission
- Request admission exposed at pass start.
- Cache
Tier Attribution - First-admission provenance for prompt tokens reused across cache tiers.
- Command
Effects - Effects of a scheduler command.
- Engine
Config - Serializable configuration for one scheduler rank.
- Engine
Factory - Constructs engines with one process-local timing provider.
- Forward
Pass Metrics - Per-pass scheduling statistics.
- Handoff
Id - Stable identifier for one replay-local prefill-to-decode handoff attempt.
- Handoff
Transfer Timing - Source-provided inputs for calculating KV transfer delay.
- KvBlock
- One runtime-neutral KV block identity.
- KvEvent
- Ordered runtime-neutral KV event emitted by one rank.
- Metrics
- Rank-local scheduler and G1 metrics.
- Native
Host Offload Config - Physical controls for framework-native G1-to-host offload.
- Output
- One client-visible output.
- Pass
Completion Effects - Effects released at the modeled pass completion boundary.
- Pass
Start Effects - Effects visible as soon as an engine pass starts.
- Pressure
Event - One scheduler-owned KV pressure action emitted with pass-start effects.
- Pressure
State - Runtime-neutral scheduler and KV occupancy around one pressure action.
- Request
- Runtime-neutral request accepted by the rank engine.
- Running
Mean - A generic running mean calculator with a fixed-size sliding window. Maintains a running sum and count to compute the mean in O(1) time.
- Scheduler
Rank - One preserved vLLM/SGLang scheduler rank behind the neutral contract.
- Sglang
Config - Serializable SGLang scheduler controls.
- Stored
Blocks - A consecutive set of newly visible blocks.
- Trtllm
Config - Serializable TensorRT-LLM scheduler controls.
Enums§
- Backend
- Scheduler semantics selected for an AISimulate rank.
- Command
- Commands supported by the standalone scheduler.
- Command
Result - Result of applying a scheduler command.
- KvEvent
Data - Runtime-neutral KV event payload.
- Lifecycle
Event - Asynchronous scheduler lifecycle fact consumed by the Replayer’s handoff coordinator.
- Preemption
Mode - Decode preemption victim selection.
- Pressure
Kind - Scheduler action taken to relieve KV pressure.
- Sglang
Schedule Policy - SGLang waiting-queue ordering.
- Timing
Model Config - Serializable timing-provider selection.
- Transfer
Timing Mode - Prompt footprint used when the Replayer models KV transfer time.
- Trtllm
Capacity Policy - TensorRT-LLM capacity scheduler policy.
- Worker
Type - Scheduler role.
Traits§
- Timing
Model - Runtime latency model injected at the engine boundary.
Functions§
- normalize_
conditional_ accept_ rates - Parse and normalize MTP/EAGLE conditional draft-token acceptance rates.
- prefill_
handoff_ delay_ ms - Compute the client-visible prefill-to-decode handoff delay.
Type Aliases§
- Engine
- A single-rank or attention-DP engine.