Struct TokenInfo 

pub struct TokenInfo {
    pub rank: usize,
    pub token_bytes: String,
    pub token_str: Option<String>,
}

Information about a vocabulary token.

This struct contains metadata about a single token in the vocabulary, including its rank (position), byte representation, and optional string form.

Fields

• rank - Position of the token in the vocabulary (used as token ID)
• token_bytes - Base64-encoded byte representation of the token
• token_str - Optional human-readable string representation

Fields

rank: usize

The position of this token in the vocabulary (used as token ID).

token_bytes: String

Base64-encoded byte representation of the token.

token_str: Option<String>

Optional human-readable string representation of the token.

Trait Implementations

impl Clone for TokenInfo

fn clone(&self) -> TokenInfo

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenInfo

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for TokenInfo

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Serialize for TokenInfo

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.