Module bert_encoder 

BERT: the encoder graph, transcribed from llama.cpp src/models/bert.cpp (llama_model_bert::graph::graph).

Loading lives next door in crate::bert_gguf_loader; pooling in crate::pooling; the reason this is not an crate::engine::Engine in crate::encoder.

The graph, and the five places it is not a decoder

h[i] = tok_embd[t[i]] + type_embd[0] + pos_embd[i]      (1) (2)
h    = LayerNorm(h, token_embd_norm)                        (3)
for each layer:
    q,k,v = Wq h + bq,  Wk h + bk,  Wv h + bv
    a     = softmax(q·kᵀ / √head_dim) v                  (4)
    x     = LayerNorm(Wo a + bo + h,  attn_output_norm)      (3)
    f     = W_down · GELU(W_up x + b_up) + b_down        (5)
    h     = LayerNorm(f + x, layer_output_norm)              (3)
result = h                                              (6)

1. Learned position embeddings, added. Not RoPE. pos_embd is a real [n_ctx_train, n_embd] table and position i is a row lookup. A learned table cannot be extrapolated, which is why crate::encoder::EncodeError::TooLong is an error and not a warning.
2. A token-type embedding. Upstream hardcodes row 0 (“Sentence A”) — ggml_view_1d(ctx0, model.type_embd, n_embd, 0) — because the single-sequence embedding use never has a sentence B. Ferrox does the same, and the loader refuses a checkpoint whose token_types table is missing when the metadata says it should have one.
3. LayerNorm, not RMSNorm, at three sites per layer plus one on the input. Mean-subtracting, and every one of them carries a bias tensor as well as a weight. Substituting RMSNorm here loads fine and produces a plausible-looking vector that is wrong.
4. No causal mask. Row 0 attends to the last token. This is the single property that makes the whole model an encoder, and attention_is_bidirectional_not_causal below is the test that would go red if a mask ever appeared.
5. A plain GELU MLP, not a gated one. Two matrices, not three, and both carry biases. LLM_FFN_GELU, LLM_FFN_SEQ upstream.
6. No output head and no logits. The hidden states are the result (res->t_embd); this checkpoint has no output.weight at all.

What this module does not do

Only arch == "bert", and only its dense, non-RoPE, separate-QKV shape. nomic-bert (RoPE + gated FFN), jina-bert-v2 (GEGLU + a second attention norm), nomic-bert-moe (expert layers) and modern-bert all share bert.cpp upstream and are all refused by name in the loader instead of being run through this graph.

Structs

BertEncoder

BertHparams

bert.* metadata, after the loader has checked it.

BertLayer

One transformer block’s weights. Biases that llama.cpp marks TENSOR_NOT_REQUIRED are Option, so a checkpoint without them is run without them rather than with a silently fabricated zero vector.