Crate tiktoken_rust

tiktoken_rust

This crate is a tokeniser for use with OpenAI’s models.

Structs

• Encoding

Enums

• AllowedSpecial
• DecodeMode
• DisallowedSpecial
• EncodeError

Functions

• encoding_for_model
  Returns the encoding used by a model.
• get_encoding
  Return Encoding object TODO: cache created Encoding object
• list_encoding_names
  List available encodings by name

Type Definitions

• Result