aprender 0.40.1

Next-generation ML framework in pure Rust — `cargo install aprender` for the `apr` CLI
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

<!-- PCU: cli-tokenize | contract: contracts/apr-page-cli-tokenize-v1.yaml -->

# apr tokenize

Tokenizer training pipeline (plan/apply) — BPE vocabulary learning

**Category**: Training

## Synopsis

```text
apr tokenize [OPTIONS]
```

## Example

<!-- example-cost: model-required model: qwen2.5-coder-1.5b -->
```bash
apr tokenize "Hello world" --tokenizer qwen2.5-coder-1.5b
```

## Full help

Run `apr tokenize --help` for the complete option list.

## See also

- Source: [`crates/apr-cli/src/commands/tokenize.rs`](https://github.com/paiml/aprender/blob/main/crates/apr-cli/src/commands/tokenize.rs)
- Contract: [`contracts/apr-page-cli-tokenize-v1.yaml`](https://github.com/paiml/aprender/blob/main/contracts/apr-page-cli-tokenize-v1.yaml)