Struct AllocationRequest 

pub struct AllocationRequest {
    pub request_id: RequestId,
    pub initial_tokens: usize,
    pub max_sequence_length: usize,
    pub num_layers: usize,
    pub num_heads: usize,
    pub head_dim: usize,
    pub device: Device,
    pub dtype: DataType,
    pub priority: Priority,
}

KV cache allocation request

Fields

request_id: RequestId

Request ID this allocation is for

initial_tokens: usize

Initial number of tokens

max_sequence_length: usize

Maximum expected sequence length

num_layers: usize

Number of layers to cache

num_heads: usize

Number of attention heads

head_dim: usize

Head dimension

device: Device

Target device

dtype: DataType

Data type for cache

priority: Priority

Priority level for allocation

Implementations

impl AllocationRequest

pub fn estimated_memory_bytes(&self) -> usize

Calculate estimated memory requirement

Trait Implementations

impl Clone for AllocationRequest

fn clone(&self) -> AllocationRequest

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for AllocationRequest

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.