Trait TextEncoder 

pub trait TextEncoder {
    // Required methods
    fn n_embd(&self) -> usize;
    fn n_ctx_train(&self) -> usize;
    fn pooling_type(&self) -> PoolingType;
    fn encode_tokens(&self, tokens: &[u32]) -> Result<Vec<f32>, EncodeError>;

    // Provided methods
    fn wrap_special(&self, pieces: &[u32]) -> Vec<u32> { ... }
    fn wrap_special_pair(&self, _a: &[u32], _b: &[u32]) -> Option<Vec<u32>> { ... }
    fn embed_tokens(&self, tokens: &[u32]) -> Result<Vec<f32>, EncodeError> { ... }
}

A model that turns a whole token sequence into hidden states in one pass, with no carried state and no logits.

Required Methods

fn n_embd(&self) -> usize

Width of one hidden-state row, and of the pooled embedding.

fn n_ctx_train(&self) -> usize

Longest sequence this checkpoint can represent. For a learned position table this is the table’s height, and exceeding it is an error rather than a degradation.

fn pooling_type(&self) -> PoolingType

What the checkpoint’s own {arch}.pooling_type said.

fn encode_tokens(&self, tokens: &[u32]) -> Result<Vec<f32>, EncodeError>

n_tokens × n_embd hidden states, in row order.

Provided Methods

fn wrap_special(&self, pieces: &[u32]) -> Vec<u32>

Wraps a tokenizer’s pieces in whatever the model requires around them — for BERT, [CLS] … [SEP].

This is on the encoder rather than the tokenizer because ferrox’s tokenizers deliberately encode text only (they are checked token-for-token against llama_tokenize(..., add_special = false, ...)), while llama.cpp keeps add_special in the vocab and applies it here. The default adds nothing, so an encoder that genuinely needs no wrapper does not have to say so.

fn wrap_special_pair(&self, _a: &[u32], _b: &[u32]) -> Option<Vec<u32>>

The two-segment input a cross-encoder scores: one sequence holding a query and a document with the model’s own boundary between them. For BERT that is [CLS] a [SEP] b [SEP].

None — the default — means this encoder has no two-segment form, and a caller that needs one must refuse. Deliberately NOT defaulted to wrap_special(a ++ b): a cross-encoder was trained with a separator between the halves, and one that never sees it still returns a plausible float. That is the “computes something else” failure, and it is invisible — a rerank with no boundary produces an ordering, just not the model’s.

fn embed_tokens(&self, tokens: &[u32]) -> Result<Vec<f32>, EncodeError>

Self::encode_tokens followed by the checkpoint’s own pooling. Not L2-normalized — see crate::pooling::pool.

Dyn Compatibility

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors

impl TextEncoder for BertEncoder