Skip to main content Crate needle_core Copy item path Source pub use config::TransformerConfig ;attn Multi-head attention with GQA (Grouped Query Attention) and KV cache. config ffn Feed-forward network (FFN) sub-layer.
Python: gate_proj + up_proj (both [d_model, d_ff]) → activation(gate)*up → down_proj [d_ff, d_model].
Activation variants: drelu = relu(gate)*relu(up), swiglu = silu(gate)*up, geglu = gelu(gate)*up.
In the default Needle config, encoder FFN is disabled (no_feedforward=true). layers Encoder and decoder layer stacks. math Thin wrappers around libm for no_std float math. model Top-level Needle SAN (Simple Attention Network) model. norm ZCRMSNorm: zero-centred RMSNorm — (1 + γ) * x / RMS(x).
Scale γ initialized to 0, so on first forward pass this reduces to plain RMSNorm. ops Primitive tensor operations: f32 only, no allocations. quant INT4 weight quantization/dequantization, bit-for-bit parity with Python quantize.py. rope Rotary Position Embeddings (RoPE).
Applied to Q and K before the attention dot product.
Uses base=10000, split-half rotation: first half × cos − second half × sin,
second half × cos + first half × sin. Matches Python apply_rope exactly.