Struct PagedAttentionExecutor 

pub struct PagedAttentionExecutor { /* private fields */ }

A model executor that actually uses paged KV cache for attention.

Uses identity projections: for each token, the embedding is a one-hot vector of length num_kv_heads * head_dim derived from the token ID. Q = K = V = embedding. This makes attention outputs deterministic and verifiable.

Logits are produced by summing attention output elements per head and distributing across vocab positions, so different attention patterns produce different token predictions.

Implementations

impl PagedAttentionExecutor

pub fn new( config: PagedExecutorConfig, kv_manager: Arc<PagedKvCacheManager>, ) -> Self

pub fn prefill_count(&self) -> u64

pub fn decode_count(&self) -> u64

Trait Implementations

impl ModelExecutor for PagedAttentionExecutor

fn info(&self) -> &ModelInfo

Get model information and metadata

fn prefill<'life0, 'life1, 'async_trait>( &'life0 self, input: &'life1 PrefillInput, ) -> Pin<Box<dyn Future<Output = Result<PrefillOutput>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Execute prefill phase (process initial prompt)

fn decode<'life0, 'life1, 'async_trait>( &'life0 self, input: &'life1 DecodeInput, ) -> Pin<Box<dyn Future<Output = Result<DecodeOutput>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Execute decode phase (generate next token)

fn capabilities(&self) -> ExecutorCapabilities

Get executor capabilities

fn status(&self) -> ExecutorStatus

Get current executor status

fn batch_decode<'life0, 'life1, 'async_trait>( &'life0 self, inputs: &'life1 [DecodeInput], ) -> Pin<Box<dyn Future<Output = Result<Vec<DecodeOutput>, FerrumError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, Self: 'async_trait,

Batch decode: process multiple sequences in one forward pass.

fn forward<'life0, 'life1, 'async_trait>( &'life0 self, _input: &'life1 Arc<dyn TensorLike>, ) -> Pin<Box<dyn Future<Output = Result<Arc<dyn TensorLike>, FerrumError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, Self: 'async_trait,

Optional: full forward pass (for non-autoregressive use cases)

fn warmup<'life0, 'async_trait>( &'life0 mut self, ) -> Pin<Box<dyn Future<Output = Result<(), FerrumError>> + Send + 'async_trait>>

where 'life0: 'async_trait, Self: 'async_trait,

Warm up executor (load model, allocate memory, etc.)

fn shutdown<'life0, 'async_trait>( &'life0 mut self, ) -> Pin<Box<dyn Future<Output = Result<(), FerrumError>> + Send + 'async_trait>>

where 'life0: 'async_trait, Self: 'async_trait,

Shutdown executor gracefully

fn release_cache(&self, _cache_id: &str)

Release KV cache and state for a completed sequence.