Struct Qwen3ModelExecutor 

pub struct Qwen3ModelExecutor { /* private fields */ }

Candle-based Qwen3 model executor with multi-sequence support.

Each active sequence gets its own KV cache keyed by a unique cache_id. This allows concurrent prefill and decode across many sequences without one sequence’s prefill destroying another’s KV cache.

On CUDA devices, lazily creates a CudaDecodeRunner that bypasses candle for the decode hot path, using cuBLAS + custom kernels with pre-allocated buffers and optional CUDA Graph acceleration.

Implementations

impl Qwen3ModelExecutor

pub fn new(model: Qwen3ModelWrapper, info: ModelInfo) -> Self

pub fn release_sequence(&self, cache_id: &str)

Release a sequence’s KV cache, freeing GPU memory. Should be called when a request completes.

Trait Implementations

impl ModelExecutor for Qwen3ModelExecutor

fn info(&self) -> &ModelInfo

Get model information and metadata

fn prefill<'life0, 'life1, 'async_trait>( &'life0 self, input: &'life1 PrefillInput, ) -> Pin<Box<dyn Future<Output = Result<PrefillOutput>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Execute prefill phase (process initial prompt)

fn decode<'life0, 'life1, 'async_trait>( &'life0 self, input: &'life1 DecodeInput, ) -> Pin<Box<dyn Future<Output = Result<DecodeOutput>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Execute decode phase (generate next token)

fn batch_decode<'life0, 'life1, 'async_trait>( &'life0 self, inputs: &'life1 [DecodeInput], ) -> Pin<Box<dyn Future<Output = Result<Vec<DecodeOutput>>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Batch decode: process multiple sequences in one forward pass.

fn capabilities(&self) -> ExecutorCapabilities

Get executor capabilities

fn release_cache(&self, cache_id: &str)

Release KV cache and state for a completed sequence.

fn status(&self) -> ExecutorStatus

Get current executor status

fn forward<'life0, 'life1, 'async_trait>( &'life0 self, _input: &'life1 Arc<dyn TensorLike>, ) -> Pin<Box<dyn Future<Output = Result<Arc<dyn TensorLike>, FerrumError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, Self: 'async_trait,

Optional: full forward pass (for non-autoregressive use cases)

fn warmup<'life0, 'async_trait>( &'life0 mut self, ) -> Pin<Box<dyn Future<Output = Result<(), FerrumError>> + Send + 'async_trait>>

where 'life0: 'async_trait, Self: 'async_trait,

Warm up executor (load model, allocate memory, etc.)

fn shutdown<'life0, 'async_trait>( &'life0 mut self, ) -> Pin<Box<dyn Future<Output = Result<(), FerrumError>> + Send + 'async_trait>>

where 'life0: 'async_trait, Self: 'async_trait,

Shutdown executor gracefully