Struct TikToken

pub struct TikToken { /* private fields */ }

Available on crate feature tiktoken only.

A tiktoken based tokenizer for openai models. Can also be used for other models.

Implements EstimateTokens for various swiftide types (prompts, chat messages, lists of chat messages) and regular strings.

Estimates are estimates; not exact counts.

Example

let tokenizer = TikToken::try_from_model("gpt-4-0314").unwrap();
let estimate = tokenizer.estimate("hello {{world}}").await.unwrap();

assert_eq!(estimate, 4);

Implementations

impl TikToken

pub fn try_from_model(model: impl AsRef<str>) -> Result<Self>

Build a TikToken from an openai model name

Errors

Errors if the tokenizer cannot be found from the model or it cannot be build

pub fn try_from_tokenizer(tokenizer: Tokenizer) -> Result<Self>

Build a TikToken from a tiktoken_rs::tiktoken::Tokenizer

Errors

Errors if the tokenizer cannot be build

Trait Implementations

impl Clone for TikToken

fn clone(&self) -> TikToken

Returns a duplicate of the value.

const fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TikToken

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for TikToken

fn default() -> Self

Returns the “default value” for a type.

impl EstimateTokens for TikToken

fn estimate<'life0, 'async_trait>( &'life0 self, value: impl 'async_trait + Estimatable, ) -> Pin<Box<dyn Future<Output = Result<usize>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,