Struct ContinuousBatchEngine 

pub struct ContinuousBatchEngine { /* private fields */ }

Continuous batching inference engine.

Wraps an Arc<EngineInner> so it can be cloned and shared freely. Multiple concurrent infer() / infer_stream() calls are safe — an internal iteration_lock serializes engine steps while allowing all pending requests to be processed in each iteration’s batch.

Implementations

impl ContinuousBatchEngine

pub fn new( config: EngineConfig, scheduler: Arc<ContinuousBatchScheduler>, tokenizer: Arc<dyn Tokenizer + Send + Sync>, sampler: Arc<dyn Sampler + Send + Sync>, kv_cache: Arc<dyn KvCacheManager + Send + Sync>, model_executor: Arc<dyn ModelExecutor + Send + Sync>, tensor_factory: Arc<dyn TensorFactory>, ) -> Result<Self>

pub fn new_with_speculation( config: EngineConfig, scheduler: Arc<ContinuousBatchScheduler>, tokenizer: Arc<dyn Tokenizer + Send + Sync>, sampler: Arc<dyn Sampler + Send + Sync>, kv_cache: Arc<dyn KvCacheManager + Send + Sync>, model_executor: Arc<dyn ModelExecutor + Send + Sync>, tensor_factory: Arc<dyn TensorFactory>, draft_executor: Option<Arc<dyn ModelExecutor + Send + Sync>>, spec_config: Option<SpeculativeDecodingConfig>, ) -> Result<Self>

Build an engine with optional speculative decoding. Pass both the draft executor AND the config together — either both or neither.

pub fn new_with_speculation_and_recurrent_state_manager( config: EngineConfig, scheduler: Arc<ContinuousBatchScheduler>, tokenizer: Arc<dyn Tokenizer + Send + Sync>, sampler: Arc<dyn Sampler + Send + Sync>, kv_cache: Arc<dyn KvCacheManager + Send + Sync>, model_executor: Arc<dyn ModelExecutor + Send + Sync>, tensor_factory: Arc<dyn TensorFactory>, draft_executor: Option<Arc<dyn ModelExecutor + Send + Sync>>, spec_config: Option<SpeculativeDecodingConfig>, recurrent_state_manager: Option<Arc<dyn RecurrentStateManager + Send + Sync>>, ) -> Result<Self>

Build an engine with optional speculative decoding and an optional recurrent-state manager for state-space / hybrid models.

pub fn new_plan_runtime( config: EngineConfig, scheduler: Arc<ContinuousBatchScheduler>, tokenizer: Arc<dyn Tokenizer + Send + Sync>, sampler: Arc<dyn Sampler + Send + Sync>, model_executor: Arc<dyn ModelExecutor + Send + Sync>, tensor_factory: Arc<dyn TensorFactory>, ) -> Result<Self>

Build an engine bound to the shared plan runtime, which is the sole owner of request-lifetime KV, recurrent state, and backing capacity. The model executor adapts that runtime but does not own a second resource manager; no legacy engine manager is created or retained.

pub fn prefix_cache_hits(&self) -> u64

Hit count since engine construction (prefix cache). Exposed for tests + /metrics endpoint; monotonic, Relaxed-ordered.

pub fn prefix_cache_stats(&self) -> PrefixCacheStats

Snapshot of prefix cache stats (hits/misses/evictions/active entries).

pub fn start_loop(&self) -> JoinHandle<()>

Start a background iteration loop. Returns a JoinHandle that runs until shutdown() is called. When a background loop is active, infer() / infer_stream() simply submit and wait.

Trait Implementations

impl Debug for ContinuousBatchEngine

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl InferenceEngine for ContinuousBatchEngine

fn status<'life0, 'async_trait>( &'life0 self, ) -> Pin<Box<dyn Future<Output = EngineStatus> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Get current engine status.

fn shutdown<'life0, 'async_trait>( &'life0 self, ) -> Pin<Box<dyn Future<Output = Result<()>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Shutdown engine gracefully.

fn config(&self) -> &EngineConfig

Get engine configuration.

fn metrics(&self) -> EngineMetrics

Get engine metrics.

fn cache_metrics_snapshot(&self) -> Option<Value>

Optional cache metrics emitted by concrete LLM engines.

fn admission_snapshot(&self) -> Result<Option<ExecutorAdmissionSnapshot>>

Runtime-authoritative admission state. Startup sizing estimates are intentionally not accepted through this method.

fn lora_metrics_snapshot(&self) -> Option<Value>

Optional LoRA runtime metrics emitted by concrete LLM engines.

fn health_check<'life0, 'async_trait>( &'life0 self, ) -> Pin<Box<dyn Future<Output = HealthStatus> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Health check.

impl LlmInferenceEngine for ContinuousBatchEngine

fn infer<'life0, 'async_trait>( &'life0 self, request: InferenceRequest, ) -> Pin<Box<dyn Future<Output = Result<InferenceResponse>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Execute single inference request.

fn infer_stream<'life0, 'async_trait>( &'life0 self, request: InferenceRequest, ) -> Pin<Box<dyn Future<Output = Result<Pin<Box<dyn Stream<Item = Result<StreamChunk>> + Send>>>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Execute streaming inference request.