tokie 0.0.2

Blazingly fast tokenizer - 50x faster tokenization, 10x smaller model files, 100% accurate drop-in replacement for HuggingFace
Documentation 
1
2
3
4
5
6
7

<p align="center">
  <img src="tokie.png" alt="tokie">
</p>

<p align="center">
  <i>A fast, correct BPE tokenizer for Rust</i>
</p>