Module continuous_batcher 

ContinuousBatcher Implementation (PMAT-015)

Implements continuous batching for LLM inference per cbtop spec §19.

Overview

Continuous batching processes inference requests dynamically, allowing new requests to join and completed requests to leave mid-batch.

Citations

• [Yu et al. 2022] “ORCA: Continuous Batching for LLM Inference” OSDI
• [Leviathan et al. 2023] “Fast Inference from Transformers via Speculative Decoding” ICML
• [Chen et al. 2023] “Accelerating LLM Decoding with Speculative Sampling” arXiv

Structs

BatchSchedule

Batch schedule result.

BatcherStats

Batcher statistics.

ContinuousBatcher

Continuous batching scheduler for LLM inference.

ExponentialMovingAverage

Exponential moving average for tracking metrics.

InferenceRequest

Inference request.

Priority

Request priority level.

SequenceGroup

Sequence group (request + generation state).

SpeculativeDecoder

Speculative decoding coordinator.

SpeculativeOutput

Output from speculative decoding step.

TokenOutput

Token output from a decode step.

Enums

SchedulingPolicy

Scheduling policy for request prioritization.

Type Aliases

Token

Token type (simplified - u32 vocabulary index).