Struct LlmExecutor 

pub struct LlmExecutor { /* private fields */ }

Implementations

impl LlmExecutor

pub fn new(model: Box<dyn DecoderOnlyLLM>, info: ModelInfo) -> Self

pub fn truncate_kv_for_cache_id(&self, cache_id: &str, new_len: usize)

Roll the KV cache for cache_id back to new_len positions. Used by speculative decoding on partial rejection. The caller must supply a GenericKvCacheHandle whose seq_len is also updated.

Trait Implementations

impl ModelExecutor for LlmExecutor

fn batch_prefill<'life0, 'life1, 'async_trait>( &'life0 self, inputs: &'life1 [PrefillInput], ) -> Pin<Box<dyn Future<Output = Result<Vec<PrefillOutput>>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Batched prefill: combine all prompts into ONE model.unified_forward call so launch / kernel-overhead is amortized across the cohort.

Falls back to the trait default (serial per-item) when the model returns Err(unsupported) from unified_forward — e.g. Qwen3MoeModel today, until Phase 2 adds its native unified path.

fn batch_decode<'life0, 'life1, 'async_trait>( &'life0 self, inputs: &'life1 [DecodeInput], ) -> Pin<Box<dyn Future<Output = Result<Vec<DecodeOutput>>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Override default fallback to acquire the model lock ONCE for the whole batch, avoiding N round-trips through parking_lot. Does not yet do true attention batching (each cache has its own kv_len), but removes mutex churn that was serialising concurrent requests at async level.

fn unified_decode<'life0, 'life1, 'async_trait>( &'life0 self, batch: &'life1 UnifiedBatch, ) -> Pin<Box<dyn Future<Output = Result<Vec<Option<Vec<f32>>>>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Unified mixed-batch dispatch (chunked-prefill API).

This impl is a behavior-preserving FALLBACK over the existing trait methods on DecoderOnlyLLM: prefill items go through model.prefill(seq_id, &q_tokens) (one at a time, mirroring the engine’s current sequential prefill loop), decode items (q_len == 1 && is_final_chunk) are grouped into a single model.decode_batch(...) call. Net behavior is identical to the engine’s pre-Phase-13 path; this just changes WHO orchestrates the prefill/decode split (caller → unified_decode) so the engine can converge on a single call.

The real performance unlock comes in Step 5 when models override this with a true unified-forward (one [M_total, hidden] forward

• varlen attention) — at that point the kernel-level mix replaces the host-side serial dispatch here.

fn info(&self) -> &ModelInfo

Get model information and metadata

fn kv_capacity(&self) -> Option<usize>

Per-request KV capacity in tokens when the executor owns a smaller runtime cache window than the model’s declared context length.

fn prefill<'life0, 'life1, 'async_trait>( &'life0 self, input: &'life1 PrefillInput, ) -> Pin<Box<dyn Future<Output = Result<PrefillOutput>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Execute prefill phase (process initial prompt)

fn truncate_kv<'life0, 'life1, 'async_trait>( &'life0 self, kv_cache: &'life1 Arc<dyn KvCacheHandle>, new_len: usize, ) -> Pin<Box<dyn Future<Output = Result<()>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Roll the KV cache for this executor’s sequence back to new_len. Used by speculative decoding on partial rejection so the next iteration sees a KV prefix that matches the accepted token stream. Default: Ok(()) — executors that don’t cache per-sequence state (stub, mock) are inherently tolerant; real LLM executors override.

fn forward_verify<'life0, 'life1, 'async_trait>( &'life0 self, inputs: &'life1 [DecodeInput], ) -> Pin<Box<dyn Future<Output = Result<Vec<DecodeOutput>>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Multi-position decode-verify: one forward over N+1 tokens, producing one logits row per position. Used by speculative decoding’s target path so we don’t pay N+1 sequential forwards.

fn decode<'life0, 'life1, 'async_trait>( &'life0 self, input: &'life1 DecodeInput, ) -> Pin<Box<dyn Future<Output = Result<DecodeOutput>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Execute decode phase (generate next token)

fn release_cache(&self, cache_id: &str)

Release KV cache and state for a completed sequence.

fn capabilities(&self) -> ExecutorCapabilities

Get executor capabilities

fn status(&self) -> ExecutorStatus

Get current executor status

fn cache_metrics_snapshot(&self) -> Option<Value>

Optional model/executor cache metrics.

fn lora_metrics_snapshot(&self) -> Option<Value>

Optional LoRA runtime metrics.

fn forward<'life0, 'life1, 'async_trait>( &'life0 self, _input: &'life1 Arc<dyn TensorLike>, ) -> Pin<Box<dyn Future<Output = Result<Arc<dyn TensorLike>, FerrumError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, Self: 'async_trait,

Optional: full forward pass (for non-autoregressive use cases)

fn warmup<'life0, 'async_trait>( &'life0 mut self, ) -> Pin<Box<dyn Future<Output = Result<(), FerrumError>> + Send + 'async_trait>>

where 'life0: 'async_trait, Self: 'async_trait,

Warm up executor (load model, allocate memory, etc.)

fn shutdown<'life0, 'async_trait>( &'life0 mut self, ) -> Pin<Box<dyn Future<Output = Result<(), FerrumError>> + Send + 'async_trait>>

where 'life0: 'async_trait, Self: 'async_trait,

Shutdown executor gracefully