pub trait Encoder: Send + Sync {
// Required methods
fn encode(&self, input: &str) -> Result<Encoding>;
fn encode_batch(&self, inputs: &[&str]) -> Result<Vec<Encoding>>;
// Provided method
fn encode_segments(
&self,
_segments: &[EncodeSegment<'_>],
) -> Result<Encoding> { ... }
}Required Methods§
fn encode(&self, input: &str) -> Result<Encoding>
fn encode_batch(&self, inputs: &[&str]) -> Result<Vec<Encoding>>
Provided Methods§
Sourcefn encode_segments(&self, _segments: &[EncodeSegment<'_>]) -> Result<Encoding>
fn encode_segments(&self, _segments: &[EncodeSegment<'_>]) -> Result<Encoding>
Encode Kimi K3-style renderer segments while preserving trusted control-token and untrusted content boundaries.
Each segment controls whether added/control tokens are recognized
through EncodeSegment::allow_special. This prevents text in user,
tool, or attribute content from becoming structural when it happens
to resemble a control token.
The Baseten and TikToken backends preserve legacy tiktoken behavior
by splitting each segment into chunks of at most 400,000 characters
and splitting whitespace/non-whitespace runs at 25,000 characters.
Baseten encodes independent chunks through the Rayon thread pool,
then concatenates them in input order, and applies tokenizer
post-processing once after all segment IDs have been joined.
TikToken classifies whitespace like the reference Kimi tokenizer
(Python’s str.isspace), while Baseten uses char::is_whitespace;
the two differ only on runs of U+001C..U+001F.
Backends must not implement this by flattening the segments first, because that discards the special-token trust boundary.
Dyn Compatibility§
This trait is dyn compatible.
In older versions of Rust, dyn compatibility was called "object safety".