Skip to main content

Encoder

Trait Encoder 

Source
pub trait Encoder: Send + Sync {
    // Required methods
    fn encode(&self, input: &str) -> Result<Encoding>;
    fn encode_batch(&self, inputs: &[&str]) -> Result<Vec<Encoding>>;

    // Provided method
    fn encode_segments(
        &self,
        _segments: &[EncodeSegment<'_>],
    ) -> Result<Encoding> { ... }
}

Required Methods§

Source

fn encode(&self, input: &str) -> Result<Encoding>

Source

fn encode_batch(&self, inputs: &[&str]) -> Result<Vec<Encoding>>

Provided Methods§

Source

fn encode_segments(&self, _segments: &[EncodeSegment<'_>]) -> Result<Encoding>

Encode Kimi K3-style renderer segments while preserving trusted control-token and untrusted content boundaries.

Each segment controls whether added/control tokens are recognized through EncodeSegment::allow_special. This prevents text in user, tool, or attribute content from becoming structural when it happens to resemble a control token.

The Baseten and TikToken backends preserve legacy tiktoken behavior by splitting each segment into chunks of at most 400,000 characters and splitting whitespace/non-whitespace runs at 25,000 characters. Baseten encodes independent chunks through the Rayon thread pool, then concatenates them in input order, and applies tokenizer post-processing once after all segment IDs have been joined. TikToken classifies whitespace like the reference Kimi tokenizer (Python’s str.isspace), while Baseten uses char::is_whitespace; the two differ only on runs of U+001C..U+001F.

Backends must not implement this by flattening the segments first, because that discards the special-token trust boundary.

Dyn Compatibility§

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§