bunsen 0.42.0

bunsen is a batteries included common library for burn
Documentation
//! # Attention operations
//!
//! Attention math and the cache state it carries, as functions and plain
//! values. Nothing here owns trainable weights: a function either borrows
//! burn's [`MultiHeadAttention`] module and uses its `Linear` layers, or takes
//! `q`, `k` and `v` tensors that are already projected.
//!
//! ## Two attention stacks
//!
//! bunsen has two attention stacks, and this module serves both.
//!
//! - **Over burn's `MultiHeadAttention`** (the Whisper kit): [`self_attn`] and
//!   [`cross_attn`], their `_w_kv_cache` variants, [`attend_q_kv_mask`] and
//!   [`causal_mask`], over a [`KVCache`]. The model keeps burn's module, so its
//!   weights, loading and any cross-checks are unchanged; only the forward path
//!   differs.
//! - **bunsen's [`CausalSelfAttention`]** block (the `NanoChat` and Qwen3 GPT
//!   kits): its own bias-free projections, QK-norm, rotary embeddings and
//!   grouped-query attention, computed with [`scaled_dot_product_attention`]
//!   over a [`KVCache`]; or, bidirectional, without one, over a padded batch's
//!   valid keys (the `Nemotron-3-Diarization` encoder).
//!
//! Use `CausalSelfAttention` for a decoder-only model, or a rotary encoder,
//! whose layers you define. Use the `MultiHeadAttention` functions when the
//! model is built on burn's module (a ported checkpoint in that layout, or
//! encoder-decoder cross-attention) and you want KV-cached decoding without
//! replacing the module.
//!
//! ## Relative positions
//!
//! [`rel_position_table`] and [`rel_shift`] are the two pieces of
//! Transformer-XL relative-position attention that are not weights: the
//! sinusoidal table of relative positions, and the shift that aligns
//! query-by-position scores into query-by-key scores. The Conformer's
//! [`RelPosSelfAttention`](crate::blocks::transformers::attention::rel_pos::RelPosSelfAttention)
//! block is built on them.
//!
//! [`length_mask`] is the `[batch, time]` validity mask of a padded batch;
//! [`window_block_mask`] is the `[n, n]` mask of block-diagonal (windowed)
//! self-attention.
//!
//! ## The cache
//!
//! Every decoder decodes through one [`KVCache`], and it is *injected*: the
//! caller builds one per decode and passes it in, and the model never owns
//! it, so one model can run several decodes in the same process. A model
//! hands out one sized for it ([`KVCacheModel`]).
//!
//! It holds each layer's self-attention history, stored as its config's
//! [`KVCacheStorage`] policy says, and, for an encoder-decoder, each layer's
//! cross-attention keys and values: projected once from the encoder output
//! ([`project_kv`]), set once, and shared by a group of decode rows (a beam
//! search's beams) rather than repeated per row.
//! [`insert_kv`](KVCache::insert_kv) appends one layer's keys and values and
//! returns that layer's full history; the position advances after the last
//! layer. A decode moves the position back without reallocating:
//! [`truncate`](KVCache::truncate) to keep a shared prefix,
//! [`reset`](KVCache::reset) to start over; and
//! [`reorder`](KVCache::reorder) permutes its rows as beams branch. It is a
//! `Module` over bare tensors: state, not parameters.
//!
//! ## Why not `MhaCache`
//!
//! The `MultiHeadAttention` functions keep a genuine KV cache: keys and
//! values are kept **projected and head-split**, so a decode step attends
//! over the whole history while projecting only the new tokens, and the
//! attention itself only scores the new queries.
//!
//! [`MhaCache`](burn::nn::attention::MhaCache) caches the *projections* and
//! expects the whole sequence back on every call, so its attention still scores
//! the full prefix each step. That is a real saving for cross-attention, whose
//! keys and values never change, but it leaves self-attention quadratic. These
//! functions take only the new tokens.
//!
//! ## Matching the uncached path
//!
//! The arithmetic mirrors `MultiHeadAttention::forward` exactly — the same
//! `1/sqrt(d_k)` scaling, the same `min_float` mask fill, the same
//! `quiet_softmax` switch — so a cached decode reproduces an uncached one.
//! `self_attn_w_kv_cache`'s contract test is what holds that true.
//!
//! Note this path deliberately omits the attention-score dropout that
//! `MultiHeadAttention::attn_scores` applies: caching is an inference concern,
//! and burn's `Dropout` is a no-op outside training anyway.
//!
//! [`MultiHeadAttention`]: burn::nn::attention::MultiHeadAttention
//! [`CausalSelfAttention`]: crate::blocks::transformers::attention::csa::CausalSelfAttention

mod attend;
mod causal_mask;
mod kv_cache;
mod length_mask;
mod rel_pos;
mod sdpa;

pub use attend::*;
pub use causal_mask::*;
pub use kv_cache::*;
pub use length_mask::*;
pub use rel_pos::*;
pub use sdpa::*;