aft-tokenizer 0.28.2

Claude lookup-encoding tokenizer for Agent File Tools
Documentation 
1
2
3
4
5

//! Claude tokenizer matching `ai-tokenizer`'s lookup/BPE hybrid encoding.

mod claude;

pub use claude::{count_tokens, encode};