Struct TokenEstimate 

pub struct TokenEstimate {
    pub gpt4: u32,
    pub claude: u32,
    pub llama: u32,
    pub generic: u32,
}

Token estimation with model awareness

Fields

gpt4: u32

Estimated tokens for GPT-4 tokenizer

claude: u32

Estimated tokens for Claude tokenizer

llama: u32

Estimated tokens for Llama tokenizer

generic: u32

Generic estimate (average)

Implementations

impl TokenEstimate

pub fn new(gpt4: u32, claude: u32, llama: u32) -> Self

Create a new token estimate with all models

pub fn compute(content: &Content) -> Self

Compute token estimate from content

pub fn for_model(&self, model: TokenModel) -> u32

Get estimate for a specific model

Trait Implementations

impl Clone for TokenEstimate

fn clone(&self) -> TokenEstimate

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenEstimate

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for TokenEstimate

fn default() -> Self

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for TokenEstimate

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl PartialEq for TokenEstimate

fn eq(&self, other: &TokenEstimate) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Serialize for TokenEstimate

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl Copy for TokenEstimate

impl Eq for TokenEstimate

impl StructuralPartialEq for TokenEstimate