token-count 0.4.0

Count tokens for LLM models using exact tokenization
Documentation
1
2
3
4
5
Hello δΈ–η•Œ 🌍
Japanese: こんにけは
Arabic: Ω…Ψ±Ψ­Ψ¨Ψ§
Emoji: πŸŽ‰πŸŽŠπŸŽˆ
Mixed: test 123 ζ΅‹θ―•