1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
//! # Tokens
//!
//! The token side of a model kit: what is shared between kits that emit ids
//! and want text back, and kits that build prompts from text. Two seams,
//! [`Detokenizer`] (ids to text) and [`Tokenizer`] (text to ids, and back);
//! an implementation of each, behind the `tokenizer` feature:
//! `WordchipperDetokenizer`, decode-only, and `BunsenTokenizer`, both ways;
//! and the parser of the `.tiktoken` rank files a
//! byte-level vocabulary ships as, [`TiktokenRanks`].
//!
//! A model kit owns its own token *layout* — which ids are special, what they
//! mean, how a prompt is built — because that is model-specific and needs no
//! dependency; Whisper's is [`WhisperTokenLayout`]. What it does not own is
//! the tokenizer. Ids-to-text is the same operation for every byte-level
//! vocabulary, and the crate that does it well is `wordchipper`, so that
//! lives here once and a kit hands it a table: Whisper's layout spells its
//! specials after the base ranks and hands the whole `{ id -> bytes }`
//! table to `WordchipperDetokenizer::from_spans`.
//!
//! Encoding — the merge table, the pre-tokenizer, the special-token
//! splitter — is a tokenizer library's whole job, and `BunsenTokenizer`
//! leaves it to the `tokenizers` crate, `transformers`' own encoder, so it
//! encodes as `transformers` does by construction. It reads a
//! `tokenizer.json`, with the added tokens of a `tokenizer_config.json` as
//! `transformers` loads them; or a pickled `tiktoken` `Encoding` (nanochat's
//! `tokenizer.pkl`, through `serde_pickle`), as the `tokenizer.json`
//! `transformers`' tiktoken converter writes for it. A kit that builds
//! prompts takes any [`Tokenizer`].
//!
//! `testing` (feature `testing`) has toy tokenizer files, in the shapes real
//! checkpoints ship, for tests of code that reads text through one.
//!
//! [`ByteLevelVocabulary`] reads the other common spelling of a byte-level
//! vocabulary, `transformers`' `vocab.json` / `tokenizer.json` (GPT-2's, and
//! Qwen's), into the same `{ id -> bytes }` table, and
//! [`TekkenVocabulary`] reads Mistral's, `tekken.json`.
//!
//! # Whisper-specific helpers
//!
//! [`SYMBOLS`], [`MISCELLANEOUS`], [`TiktokenRanks::non_speech_tokens`] and
//! [`TiktokenRanks::blank_token`] are not shared: they compute Whisper's
//! default suppress list and blank token from its rank file, and only the
//! Whisper kit's logit filters use them. They live here beside the rank parser
//! they read.
//!
//! [`WhisperTokenLayout`]: crate::kits::speech::asr::whisper::driver::WhisperTokenLayout
use Debug;
use crateBunsenResult;
pub use WordchipperDetokenizer;
pub use *;
pub use *;
pub use *;
pub use *;
/// Turns token ids into text.
///
/// The seam a kit holds as `Option<Arc<dyn Detokenizer>>`: without one, its
/// output is ids, which is still a complete result. Ids are `i64`, as the
/// model's token tensors are; the implementation narrows them.
///
/// `Debug` is a supertrait so that a module holding one can still derive
/// `Debug`; `Send + Sync` so that one can be shared across streams.
/// A checkpoint's tokenizer, text to ids and back: what a kit that builds
/// its own prompts, or holds a conversation, reads and writes text through.
///
/// Ids are `i64`, as the model's token tensors are. It is a [`Detokenizer`],
/// and shares its supertraits for the same reasons.