Skip to main content

Module mistral

Module mistral 

Source
Expand description

Mistral Model V2 - Clean implementation using solid abstractions

This implements the Mistral architecture which is used by:

  • Mistral-7B, Mistral-7B-Instruct
  • Key features: Sliding window attention, GQA
  • Uses unified Tensor type from tensor_core
  • Implements Model trait from model_core
  • Supports loading via weight_loader_core

Structsยง

MistralAttention
Mistral attention mechanism with sliding window
MistralConfig
MistralLayer
Mistral transformer layer
MistralMLP
Mistral MLP (feed-forward network)
MistralModelV2
Main Mistral model implementation