Struct TransformerMemoryPool 

pub struct TransformerMemoryPool { /* private fields */ }

Memory pool for Transformer inference

Reuses intermediate buffers to reduce allocation overhead during inference. Typical use case: allocate once, reuse across multiple forward passes.

Implementations

impl TransformerMemoryPool

pub fn new( batch_size: usize, seq_len: usize, hidden_dim: usize, num_heads: usize, ) -> Self

Create a new memory pool with specified dimensions

pub fn resize( &mut self, batch_size: usize, seq_len: usize, hidden_dim: usize, num_heads: usize, )

Update pool dimensions if needed

pub fn get_attn_score_buffer(&mut self) -> &mut Vec<f64>

Get or allocate attention score buffer

Panics

This method should never panic as it allocates the buffer if needed. Panic would only occur if memory allocation fails.

pub fn get_attn_weight_buffer(&mut self) -> &mut Vec<f64>

Get or allocate attention weight buffer

Panics

This method should never panic as it allocates the buffer if needed. Panic would only occur if memory allocation fails.

pub fn get_qkv_buffer(&mut self) -> &mut Vec<f64>

Get or allocate QKV projection buffer

Panics

This method should never panic as it allocates the buffer if needed. Panic would only occur if memory allocation fails.

pub fn get_output_buffer(&mut self) -> &mut Vec<f64>

Get or allocate output buffer

Panics

This method should never panic as it allocates the buffer if needed. Panic would only occur if memory allocation fails.

pub fn memory_bytes(&self) -> usize

Get estimated memory usage in bytes

Trait Implementations

impl Debug for TransformerMemoryPool

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for TransformerMemoryPool

fn default() -> Self

Returns the “default value” for a type.