Trait TextEncoder 

pub trait TextEncoder {
    // Required methods
    fn n_embd(&self) -> usize;
    fn n_ctx_train(&self) -> usize;
    fn pooling_type(&self) -> PoolingType;
    fn encode_tokens(&self, tokens: &[u32]) -> Result<Vec<f32>, EncodeError>;

    // Provided methods
    fn wrap_special(&self, pieces: &[u32]) -> Vec<u32> { ... }
    fn embed_tokens(&self, tokens: &[u32]) -> Result<Vec<f32>, EncodeError> { ... }
}

A model that turns a whole token sequence into hidden states in one pass, with no carried state and no logits.

Required Methods

fn n_embd(&self) -> usize

Width of one hidden-state row, and of the pooled embedding.

fn n_ctx_train(&self) -> usize

Longest sequence this checkpoint can represent. For a learned position table this is the table’s height, and exceeding it is an error rather than a degradation.

fn pooling_type(&self) -> PoolingType

What the checkpoint’s own {arch}.pooling_type said.

fn encode_tokens(&self, tokens: &[u32]) -> Result<Vec<f32>, EncodeError>

n_tokens × n_embd hidden states, in row order.

Provided Methods

fn wrap_special(&self, pieces: &[u32]) -> Vec<u32>

Wraps a tokenizer’s pieces in whatever the model requires around them — for BERT, [CLS] … [SEP].

This is on the encoder rather than the tokenizer because ferrox’s tokenizers deliberately encode text only (they are checked token-for-token against llama_tokenize(..., add_special = false, ...)), while llama.cpp keeps add_special in the vocab and applies it here. The default adds nothing, so an encoder that genuinely needs no wrapper does not have to say so.

fn embed_tokens(&self, tokens: &[u32]) -> Result<Vec<f32>, EncodeError>

Self::encode_tokens followed by the checkpoint’s own pooling. Not L2-normalized — see crate::pooling::pool.

Dyn Compatibility

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors

impl TextEncoder for BertEncoder