Crate ferrotensor_graph 

Model building blocks on top of ferrotensor-autograd: linear layers with LoRA, RMSNorm, RoPE grouped-query attention, SwiGLU MLPs, Llama and LFM2 decoders, and the fused attention layers in flash_attention.

Re-exports

pub use flash_attention::AttentionMask;

pub use flash_attention::FlashAttention;

pub use flash_attention::FlashAttentionGrads;

pub use flash_attention::GroupedQueryAttention;

pub use flash_attention::KVCache as FlashKVCache;

pub use flash_attention::MultiQueryAttention;

pub use flash_attention::SlidingWindowAttention;

pub use lfm::Lfm2Model;

Modules

flash_attention

Flash Attention - Memory-Efficient Attention Mechanism

lfm

LFM2 (Liquid foundation model) blocks.

model_builder

Model builder: constructs an Lfm2Model from a WeightsContext.

Structs

Attention

Causal grouped-query self-attention with RoPE (Llama style).

Embedding

Token embedding table.

KVCache

Per-layer KV caches of a model.

KVCacheLayer

Cached keys and values of one layer for incremental decoding.

Linear

Bias-free linear layer y = x @ weight with an optional LoRA adapter.

LlamaBlock

Pre-norm Llama decoder block: attention and MLP with residuals.

LlamaModel

A Llama-style decoder: embedding, blocks, final norm and LM head.

LoRAAdapter

Low-rank adapter added to a Linear: x @ a @ b * alpha / rank.

MLP

SwiGLU feed-forward: down(silu(gate(x)) * up(x)).

RMSNorm

Root-mean-square layer norm over the last dimension.

Functions

create_padding_mask

Helper to create a padding mask for Flash Attention V2 (additive mask)