Crate aft_tokenizer 

Claude tokenizer matching ai-tokenizer’s lookup/BPE hybrid encoding.

Functions

count_tokens

Count Claude tokens in text.

encode

Encode text into Claude token IDs.