toklab-core 0.1.1

Pure-Rust core for toklab: bulk tokenizer + counter for OpenAI BPE encodings.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

# toklab-core

Pure-Rust core for [toklab](https://github.com/MukundaKatta/toklab):
bulk tokenizer + counter for OpenAI BPE encodings, wrapping
[tiktoken-rs](https://crates.io/crates/tiktoken-rs).

```rust
use toklab_core::Tokenizer;

let tok = Tokenizer::for_model("gpt-4")?;
assert_eq!(tok.count("hello world"), 2);
let many = tok.count_many(&["hello", "world"], false);
assert_eq!(many, vec![1, 1]);
# Ok::<(), toklab_core::TokenizerError>(())
```

## License

Dual-licensed under MIT or Apache-2.0.