Skip to main content

Module tokenizer

oxideshield_core

Module tokenizer

Expand description

Tokenizer integration using tiktoken-rs

Structs§

TokenStats: Token statistics for text analysis
Tokenizer: Tokenizer for counting and encoding tokens

Enums§

TokenizerModel: Supported tokenizer models