1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
//! # Tokens
//!
//! The token side of a model kit: what is shared between kits that emit ids
//! and want text back. Today that is one seam, [`Detokenizer`], and one
//! implementation of it, `WordchipperDetokenizer`, behind the `tokenizer`
//! feature, plus the parser of the `.tiktoken` rank files a byte-level
//! vocabulary ships as, [`TiktokenRanks`].
//!
//! A model kit owns its own token *layout* — which ids are special, what they
//! mean, how a prompt is built — because that is model-specific and needs no
//! dependency; Whisper's is [`WhisperTokenLayout`]. What it does not own is
//! the tokenizer. Ids-to-text is the same operation for every byte-level
//! vocabulary, and the crate that does it well is `wordchipper`, so that
//! lives here once and a kit hands it a table: Whisper's layout spells its
//! specials after the base ranks and hands the whole `{ id -> bytes }`
//! table to `WordchipperDetokenizer::from_spans`.
//!
//! Only decoding is here. Encoding — the merge table, the pre-tokenizer, the
//! special-token splitter — is `wordchipper`'s whole job, and a kit that needs
//! it should use that crate directly.
//!
//! # Whisper-specific helpers
//!
//! [`SYMBOLS`], [`MISCELLANEOUS`], [`non_speech_tokens`] and
//! [`blank_token`] are not shared: they compute Whisper's default suppress
//! list and blank token from its rank file, and only the Whisper kit's
//! logit filters use them. They live here beside the rank parser they
//! read.
//!
//! [`WhisperTokenLayout`]: crate::kits::speech::whisper::driver::WhisperTokenLayout
use Debug;
use crateBunsenResult;
pub use WordchipperDetokenizer;
pub use *;
/// Turns token ids into text.
///
/// The seam a kit holds as `Option<Arc<dyn Detokenizer>>`: without one, its
/// output is ids, which is still a complete result. Ids are `i64`, as the
/// model's token tensors are; the implementation narrows them.
///
/// `Debug` is a supertrait so that a module holding one can still derive
/// `Debug`; `Send + Sync` so that one can be shared across streams.