Struct ScratchSpaceManager 

pub struct ScratchSpaceManager { /* private fields */ }

Manager for per-thread scratch space (non-WASM version).

Provides each thread with its own scratch buffer for temporary computations during inference, avoiding allocation on the hot path.

Design

• Each thread gets a dedicated scratch buffer on first access
• Buffers are sized based on model dimensions
• Scratch is reset at the start of each generation step
• Thread-safe lazy initialization

Implementations

impl ScratchSpaceManager

pub fn new(scratch_size: usize, max_threads: usize) -> Result<Self>

Create a new scratch space manager.

Arguments

• scratch_size - Size of each thread’s scratch buffer in bytes
• max_threads - Maximum number of threads to support

Note

Memory is allocated lazily when get_scratch is called. This method always succeeds but returns Result for API consistency with WASM.

pub fn for_model(hidden_dim: usize, max_threads: usize) -> Result<Self>

Create a scratch manager sized for model dimensions.

Arguments

• hidden_dim - Model hidden dimension
• max_threads - Maximum number of threads

pub fn get_scratch(&self) -> Result<ScratchSpace<'_>>

Get the scratch buffer for the current thread.

Creates a new buffer if this is the first access from this thread.

Returns

A reference to the thread’s scratch space.

Errors

Returns an error if the maximum thread count is exceeded or memory allocation fails.

pub fn reset_all(&self)

Reset all thread scratch buffers.

Should be called at the start of each generation step.

pub fn scratch_size(&self) -> usize

Get the configured scratch size per thread.

pub fn active_threads(&self) -> usize

Get the number of active threads with scratch buffers.

pub fn stats(&self) -> ScratchStats

Get statistics about scratch usage.

Trait Implementations

impl Debug for ScratchSpaceManager

Available on non-WebAssembly only.

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Send for ScratchSpaceManager

Available on non-WebAssembly only.

impl Sync for ScratchSpaceManager

Available on non-WebAssembly only.