Enum Encoding 

pub enum Encoding {
    Hf(Box<Encoding>),
    Sp(Vec<TokenIdType>),
    Tiktoken(Vec<TokenIdType>),
}

Contains the results of tokenizing text: token IDs, string tokens, and their spans

Variants

Hf(Box<Encoding>)

Hugging Face

Sp(Vec<TokenIdType>)

Sentence Piece

Tiktoken(Vec<TokenIdType>)

Tiktoken (for GPT models) - now uses u32 in tiktoken-rs 0.7.0

Implementations

impl Encoding

pub fn token_ids(&self) -> &[TokenIdType]

Returns a reference to token IDs - zero-copy operation

pub fn token_ids_ref(&self) -> &[TokenIdType]

👎Deprecated since 0.1.0: Use token_ids() instead

Deprecated: Use token_ids() instead (kept for compatibility)

pub fn get_hash(&self) -> u64

Get a hash of the token IDs for caching purposes

Trait Implementations

impl Clone for Encoding

fn clone(&self) -> Encoding

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Encoding

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Hash for Encoding

Hash implementation for Encoding

fn hash<H: Hasher>(&self, state: &mut H)

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H)

where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher.