Module bert_encoder 

BERT: the encoder graph, transcribed from llama.cpp src/models/bert.cpp (llama_model_bert::graph::graph).

Loading lives next door in crate::bert_gguf_loader; pooling in crate::pooling; the reason this is not an crate::engine::Engine in crate::encoder.

The graph, and the five places it is not a decoder

h[i] = tok_embd[t[i]] + type_embd[seg[i]] + pos_embd[i] (1) (2)
h    = LayerNorm(h, token_embd_norm)                        (3)
for each layer:
    q,k,v = Wq h + bq,  Wk h + bk,  Wv h + bv
    a     = softmax(q·kᵀ / √head_dim) v                  (4)
    x     = LayerNorm(Wo a + bo + h,  attn_output_norm)      (3)
    f     = W_down · GELU(W_up x + b_up) + b_down        (5)
    h     = LayerNorm(f + x, layer_output_norm)              (3)
result = h                                              (6)

1. Learned position embeddings, added. Not RoPE. pos_embd is a real [n_ctx_train, n_embd] table and position i is a row lookup. A learned table cannot be extrapolated, which is why crate::encoder::EncodeError::TooLong is an error and not a warning.
2. A token-type embedding, per position. A single-sequence embedding pass is all “Sentence A” and uses row 0, which is what upstream hardcodes — ggml_view_1d(ctx0, model.type_embd, n_embd, 0), with the comment that token types are hardcoded to zero because llama_batch carries no segment ids. A cross-encoder PAIR is not that case: HuggingFace’s tokenizer(query, document) emits 0…0 1…1 and BertModel adds row 1 to every position after the first [SEP]. Frink adds the row the caller names, which is row 0 for every embedding request and 0/1 for a rerank pair. Matching upstream here instead was measured, on cross-encoder/ms-marco-MiniLM-L6-v2 against a NumPy transcription of BertForSequenceClassification, to put the RELEVANT document LAST in three of four rankings — see tests/rerank_cross_encoder_ordering.rs.
3. LayerNorm, not RMSNorm, at three sites per layer plus one on the input. Mean-subtracting, and every one of them carries a bias tensor as well as a weight. Substituting RMSNorm here loads fine and produces a plausible-looking vector that is wrong.
4. No causal mask. Row 0 attends to the last token. This is the single property that makes the whole model an encoder, and attention_is_bidirectional_not_causal below is the test that would go red if a mask ever appeared.
5. A plain GELU MLP, not a gated one. Two matrices, not three, and both carry biases. LLM_FFN_GELU, LLM_FFN_SEQ upstream.
6. No output head and no logits. The hidden states are the result (res->t_embd); this checkpoint has no output.weight at all.

What this module does not do

Only arch == "bert", and only its dense, non-RoPE, separate-QKV shape. nomic-bert (RoPE + gated FFN), jina-bert-v2 (GEGLU + a second attention norm), nomic-bert-moe (expert layers) and modern-bert all share bert.cpp upstream and are all refused by name in the loader instead of being run through this graph.

Structs

BertEncoder

BertHparams

bert.* metadata, after the loader has checked it.

BertLayer

QkLayerNorm

One layer’s Q/K LayerNorm pair, weights and biases.

Enums

BertFfn

The two FFN shapes bert.cpp builds on this graph for the architectures frink serves.

BertTopology

One transformer block’s weights. Biases that llama.cpp marks TENSOR_NOT_REQUIRED are Option, so a checkpoint without them is run without them rather than with a silently fabricated zero vector. Where an encoder layer’s norms sit, and which function they are.