aprender 0.40.1

Next-generation ML framework in pure Rust — `cargo install aprender` for the `apr` CLI
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85

<!-- PCU: lib-text | contract: contracts/apr-page-lib-text-v1.yaml -->

# Module: `aprender::text`

Public module of the `aprender-core` crate.

## Source

[`crates/aprender-core/src/text.rs`](https://github.com/paiml/aprender/blob/main/crates/aprender-core/src/text.rs) or directory.

## Example

<!-- example-cost: skip -->
```rust
use aprender::text::{Tokenizer, ChatTemplateEngine, ChatMessage};
// See `cargo doc -p aprender-core --open` for full API reference.
```

## Module summary

`aprender::text` is the NLP toolkit. It owns the `Tokenizer` trait, BPE and
Llama-style tokenizers, the chat-template engine that turns
`Vec<ChatMessage>` into prompt strings for ChatML / Llama2 / Mistral / Phi /
Alpaca, plus classical NLP utilities — stop words, stemming, sentiment,
similarity, IDF, summarisation, vectorisation, topic modelling, and a small
RAG retrieval submodule. Anything that converts characters to tokens or
templates a conversation runs through here.

## Key types

| Type | Description |
|------|-------------|
| `Tokenizer` | Trait. `encode(&str) -> Vec<u32>`, `decode(&[u32]) -> String`. |
| `ChatTemplateEngine` | Multi-format chat-template renderer (minijinja under the hood). |
| `ChatMessage`, `SpecialTokens`, `TemplateFormat` | Building blocks for templating. |
| `ChatMLTemplate`, `Llama2Template`, `MistralTemplate`, `PhiTemplate`, `AlpacaTemplate`, `HuggingFaceTemplate`, `RawTemplate` | Concrete template implementations. |
| `auto_detect_template`, `detect_format_from_name`, `create_template` | Convenience constructors. |
| `text::bpe`, `text::llama_tokenizer`, `text::stem`, `text::similarity`, `text::vectorize`, `text::rag` | Sub-modules for specific tasks. |

## Usage patterns

### Pattern 1: Detect a chat template by model name

<!-- example-cost: skip -->
```rust
use aprender::text::{detect_format_from_name, TemplateFormat};

let fmt = detect_format_from_name("Qwen/Qwen2.5-Coder-7B-Instruct");
assert!(matches!(fmt, Some(TemplateFormat::ChatML) | Some(TemplateFormat::HuggingFace)));

let other = detect_format_from_name("meta-llama/Llama-2-7b-chat-hf");
assert!(matches!(other, Some(TemplateFormat::Llama2)));
```

### Pattern 2: Render a multi-turn conversation

<!-- example-cost: skip -->
```rust
use aprender::text::{ChatMessage, ChatMLTemplate, ChatTemplateEngine};

let template = ChatMLTemplate::default();
let messages = vec![
    ChatMessage::system("You are a helpful coding assistant."),
    ChatMessage::user("What is 2 + 2?"),
    ChatMessage::assistant("4"),
    ChatMessage::user("And 3 + 3?"),
];

let prompt = template.render(&messages, true).expect("render");
println!("--- prompt ---\n{}", prompt);
assert!(prompt.contains("system"));
assert!(prompt.contains("user"));
```

## See also

- [`models`](models.md) — Qwen2 / BERT consume tokens produced here
- [`embed`](embed.md) — vectorisation / embedding pipelines built on top
- [`code`](code.md) — code-aware tokenisation and parsing
- [`stack`](stack.md) — higher-level orchestration that bundles text + model + template

## Full API

Run `cargo doc -p aprender-core --open` for the rendered rustdoc, or browse
[docs.rs/aprender](https://docs.rs/aprender) for the published version.