aprender 0.29.3

Next-generation ML framework in pure Rust — `cargo install aprender` for the `apr` CLI
Documentation
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# BPE Tokenizer Benchmark

Loads a HuggingFace `tokenizer.json` and measures encode throughput.

Defaults to `./tokenizer.json` (Qwen2.5 shipped in repo root).

## Run

```bash
cargo run --release --example bench_bpe [-- /path/to/tokenizer.json]
```

## Source

```rust,ignore
{{#include ../../../examples/bench_bpe.rs}}
```