Skip to main content

FastTokenizer

Struct FastTokenizer 

Source
pub struct FastTokenizer { /* private fields */ }
Expand description

Hybrid tokenizer: fast BPE encoding via fastokens, decoding via HuggingFace.

Both backends are loaded from the same tokenizer.json file.

Implementations§

Source§

impl FastTokenizer

Source

pub fn from_file(path: &str) -> Result<Self>

Load tokenizer.json, ignoring its truncation and padding blocks as Python transformers does unless a call asks for them. fastokens would apply both on every encode, and CachedTokenizer would then apply them to each separately encoded piece instead of the whole input.

Trait Implementations§

Source§

impl Decoder for FastTokenizer

Source§

fn has_unstable_suffix( &self, token_ids: &[TokenIdType], skip_special_tokens: bool, ) -> bool

Whether appending tokens can still reinterpret the decoded suffix. Callers must buffer an unstable suffix until a boundary or end of input.
Source§

fn decode( &self, token_ids: &[TokenIdType], skip_special_tokens: bool, ) -> Result<DecodeResult>

Source§

impl Encoder for FastTokenizer

Source§

fn encode(&self, input: &str) -> Result<Encoding>

Source§

fn encode_batch(&self, inputs: &[&str]) -> Result<Vec<Encoding>>

Source§

fn encode_segments(&self, segments: &[EncodeSegment<'_>]) -> Result<Encoding>

Encode Kimi K3-style renderer segments while preserving trusted control-token and untrusted content boundaries. Read more
Source§

impl Tokenizer for FastTokenizer

Source§

fn validate_prefix_cache(&self) -> Result<()>

Validate that this tokenizer can be safely wrapped in the prefix cache. Read more
Source§

fn validate_segment_prefix_cache(&self) -> Result<()>

Validate that a segment which allows special tokens encodes to the ids Encoder::encode gives its text, so the prefix cache can serve it. Read more
Source§

fn vocab_size(&self) -> Option<usize>

Vocabulary cardinality including added tokens, when the backend can expose one. None for backends without a bounded id space or vocabulary introspection.
Source§

fn token_to_id(&self, token: &str) -> Result<Option<TokenIdType>>

Resolve a token string to its vocabulary id, when the backend supports lookup. Ok(None) when the token is not in the vocabulary. Err when this backend cannot do id lookup at all — kept distinct from a genuine vocabulary miss so callers can tell “unsupported” from “looked up, not found”. Read more
Source§

fn special_token_ids(&self) -> Result<Vec<TokenIdType>>

Ids of added tokens marked special (e.g. BOS/EOS/PAD/control tokens), as distinct from ordinary vocabulary tokens. Ok(vec![]) for backends that genuinely have none, or that do not distinguish special from ordinary vocabulary ids. Err when the backend cannot enumerate added tokens at all — an empty Vec alone cannot carry that distinction. Read more
Source§

fn num_special_tokens_added(&self) -> Result<usize>

Count of special tokens encode’s add_special_tokens: true path would add to a bare encoding (e.g. BOS/EOS), available without performing an encode. Ok(0) for backends that genuinely add none. Err when the backend cannot determine this at all — a plain 0 alone cannot carry that distinction, and this value feeds token-budget accounting where a silent 0 would under-count rather than fail loudly. Read more
Source§

fn with_options(self, options: TokenizerOptions) -> Self
where Self: Sized,

Apply construction-time TokenizerOptions. Read more

Auto Trait Implementations§

Blanket Implementations§

Source§

impl<T> Any for T
where T: 'static + ?Sized,

Source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
Source§

impl<T> Borrow<T> for T
where T: ?Sized,

Source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
Source§

impl<T> From<T> for T

Source§

fn from(t: T) -> T

Returns the argument unchanged.

Source§

impl<T> Instrument for T

Source§

fn instrument(self, span: Span) -> Instrumented<Self> ⓘ

Instruments this type with the provided Span, returning an Instrumented wrapper. Read more
Source§

fn in_current_span(self) -> Instrumented<Self> ⓘ

Instruments this type with the current Span, returning an Instrumented wrapper. Read more
Source§

impl<T, U> Into<U> for T
where U: From<T>,

Source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Source§

impl<T> IntoEither for T

Source§

fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ

Converts self into a Left variant of Either<Self, Self> if into_left is true. Converts self into a Right variant of Either<Self, Self> otherwise. Read more
Source§

fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
where F: FnOnce(&Self) -> bool,

Converts self into a Left variant of Either<Self, Self> if into_left(&self) returns true. Converts self into a Right variant of Either<Self, Self> otherwise. Read more
Source§

impl<T> Pointable for T

Source§

const ALIGN: usize

The alignment of pointer.
Source§

type Init = T

The type for initializers.
Source§

unsafe fn init(init: <T as Pointable>::Init) -> usize

Initializes a with the given initializer. Read more
Source§

unsafe fn deref<'a>(ptr: usize) -> &'a T

Dereferences the given pointer. Read more
Source§

unsafe fn deref_mut<'a>(ptr: usize) -> &'a mut T

Mutably dereferences the given pointer. Read more
Source§

unsafe fn drop(ptr: usize)

Drops the object pointed to by the given pointer. Read more
Source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

type Error = !

The type returned in the event of a conversion error.
Source§

fn try_from(value: U) -> Result<T, !>

Performs the conversion.
Source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.
Source§

impl<V, T> VZip<V> for T
where V: MultiLane<T>,

Source§

fn vzip(self) -> V

Source§

impl<T> WithSubscriber for T

Source§

fn with_subscriber<S>(self, subscriber: S) -> WithDispatch<Self> ⓘ
where S: Into<Dispatch>,

Attaches the provided Subscriber to this type, returning a WithDispatch wrapper. Read more
Source§

fn with_current_subscriber(self) -> WithDispatch<Self> ⓘ

Attaches the current default Subscriber to this type, returning a WithDispatch wrapper. Read more