Crate concision_ext

An extension of the concision library, focusing on providing additional layers and other non-essential features for building more complex neural networks and machine learning models.

Features

• attention: Enables various attention mechanisms from scaled dot-product and multi-headed attention to FFT-based attention.

Modules

attention

Attention

prelude

Structs

MultiHeadAttention

Multi-Headed attention is the first evolution of the Scaled Dot-Product Attention mechanism. They allow the model to jointly attend to information from different representation subspaces at different positions.

QkvParamsBase

This object is designed to store the parameters of the QKV (Query, Key, Value)

ScaledDotProductAttention

Scaled Dot-Product Attention mechanism is the core of the Transformer architecture. It computes the attention scores using the dot product of the query and key vectors, scales them by the square root of the dimension of the key vectors, and applies a softmax function to obtain the attention weights.