Skip to main content

paged_write

Function paged_write 

Source
pub fn paged_write(cache: &Array, new: &Array, slots: &Array) -> Array
Expand description

cache ([blocks, kv_heads, block_size, D]) with the positions new holds ([T, kv_heads, D], or [B, T, kv_heads, D] for several sequences) written into their slots: slots (integers, new’s leading shape) holds block slot / block_size, position slot % block_size of each. Slots must be distinct; out-of-range slots (negative, or at least blocks · block_size) are skipped. In place when the evaluation owns the cache; on the GPU, new is read through its strides (a transposed view of a contiguous cache needs no copy).