Expand description
§DeepStrike Tokenizer
Multi-model token counting engine.
Structs§
- Tokenizer
- Token counter with cached BPE instance.
Enums§
- Tokenizer
Backend - Supported tokenizer backends.
Functions§
- count_
tokens - Quick helper: count tokens with cl100k_base.