Enum BertFfn 

pub enum BertFfn {
    GeluSeq,
    SwigluPar,
    GegluPar,
    GegluFusedUp,
    SwigluFusedUp,
}

The two FFN shapes bert.cpp builds on this graph for the architectures frink serves.

bert and nomic-bert-moe’s dense layers take the ungated GELU with both biases (:179-187); nomic-bert takes the gated SiLU with none (:195-201, the final else). The variant is the architecture’s, read once at load, so a layer body cannot ask “is there a gate tensor” and answer differently on two files.

Variants

GeluSeq

down(gelu(up(x))), biases where the file has them.

SwigluPar

down(silu(gate(x)) * up(x)), no biases.

GegluPar

down(gelu(gate(x)) * up(x)): jina-bert-v2 with a separate ffn_gate (bert.cpp:188-194 picks LLM_FFN_GELU under LLM_FFN_PAR, which build_ffn turns into ggml_geglu_split).

GegluFusedUp

The same function with the gate FUSED into ffn_up: the matrix is 2 * n_ff rows and ggml_geglu splits it, the FIRST half being the gate (bert.cpp:189, up_contains_gate).

SwigluFusedUp

SwiGLU with the gate fused into ffn_up the same way: neo-bert.cpp:35,110-115 creates a 2 * n_ff-wide matrix and passes LLM_FFN_SWIGLU under LLM_FFN_SEQ, which is ggml_swiglu over the doubled row.

Trait Implementations

impl Clone for BertFfn

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for BertFfn

impl Debug for BertFfn

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for BertFfn

impl PartialEq for BertFfn

fn eq(&self, other: &Self) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for BertFfn