Crate tiktoken_rust

Source
Expand description

§tiktoken_rust

This crate is a tokeniser for use with OpenAI’s models.

Structs§

Encoding

Enums§

AllowedSpecial
DecodeMode
DisallowedSpecial
EncodeError

Functions§

encoding_for_model
Returns the encoding used by a model.
get_encoding
Return Encoding object TODO: cache created Encoding object
list_encoding_names
List available encodings by name

Type Aliases§

Result