1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
//! # Tekken - Rust Implementation of Mistral's Multimodal Tokenizer
//!
//! `tekken` is a Rust implementation of Mistral's Tekken tokenizer with full support
//! for text, image and audio tokenization. It provides high-performance, memory-safe
//! tokenization that is fully compatible with the Python implementation.
//!
//! ## Features
//!
//! - **Text Tokenization**: Full BPE (Byte Pair Encoding) support with special tokens
//! - **Image Processing**: Resize and normalize images into `[IMG]` token grids (`image` feature)
//! - **Audio Processing**: Convert audio waveforms to token sequences using mel-scale spectrograms (`audio` feature)
//! - **Multimodal Support**: Mix text, image and audio tokens in a single sequence
//! - **Version Compatibility**: Support for multiple tokenizer versions (V3, V7, V11, V13, V15)
//! - **Special Tokens**: Comprehensive handling of control, instruction, tool, and media tokens
//! - **Model Settings**: Parsing and validation of model settings constraints (V15+)
//!
//! ## Feature Flags
//!
//! Both multimodal features are enabled by default and can be turned off to
//! drop their dependencies:
//!
//! - `image` - image decoding and preprocessing, via the `image` crate
//! - `audio` - audio decoding and preprocessing, via `hound`
//!
//! Tokenizer files are parsed the same either way: [`ImageConfig`] and
//! [`AudioConfig`] are always available, as are the token layouts
//! [`ImageEncoder`] computes from image dimensions. Only decoding and
//! preprocessing actual media need the features.
//!
//! ## Quick Start
//!
//! ### Basic Text Tokenization
//!
//! ```rust,no_run
//! use tekken::{Tekkenizer, SpecialTokenPolicy};
//!
//! # fn main() -> Result<(), Box<dyn std::error::Error>> {
//! // Load tokenizer from configuration file
//! let tokenizer = Tekkenizer::from_file("tekken.json")?;
//!
//! // Encode text with BOS/EOS tokens
//! let text = "Hello, world!";
//! let tokens = tokenizer.encode(text, true, true)?;
//! println!("Tokens: {:?}", tokens);
//!
//! // Decode back to text
//! let decoded = tokenizer.decode(&tokens, SpecialTokenPolicy::Keep)?;
//! println!("Decoded: {}", decoded);
//! # Ok(())
//! # }
//! ```
//!
//! ### Image Tokenization
//!
//! ```rust,no_run
//! # #[cfg(feature = "image")]
//! # fn main() -> Result<(), Box<dyn std::error::Error>> {
//! use tekken::{Image, ImageConfig, ImageEncoder, SpecialImageIds};
//!
//! // Load an image
//! let image = Image::from_file("picture.png")?;
//!
//! // Configure image processing
//! let config = ImageConfig::new(14, 1540, 2)?;
//!
//! // Create encoder and process the image
//! let encoder = ImageEncoder::new(
//! config,
//! SpecialImageIds { img: 10, img_break: 12, img_end: 13 },
//! );
//! let encoding = encoder.encode(&image)?;
//!
//! println!("Image encoded to {} tokens", encoding.tokens.len());
//! println!("Preprocessed shape: {:?}", encoding.image.dim());
//! # Ok(())
//! # }
//! # #[cfg(not(feature = "image"))]
//! # fn main() {}
//! ```
//!
//! ### Audio Tokenization
//!
//! ```rust,no_run
//! # #[cfg(feature = "audio")]
//! # fn main() -> Result<(), Box<dyn std::error::Error>> {
//! use tekken::{Audio, AudioConfig, AudioSpectrogramConfig, AudioEncoder};
//!
//! // Load audio file
//! let audio = Audio::from_file("audio.wav")?;
//!
//! // Configure audio processing
//! let spectrogram_config = AudioSpectrogramConfig::new(80, 160, 400)?;
//! let audio_config = AudioConfig::new(16000, 12.5, spectrogram_config, None)?;
//!
//! // Create encoder and process audio
//! let encoder = AudioEncoder::new(audio_config, 1000, 1001); // audio_token_id, begin_audio_token_id
//! let encoding = encoder.encode(audio)?;
//!
//! println!("Audio encoded to {} tokens", encoding.tokens.len());
//! # Ok(())
//! # }
//! # #[cfg(not(feature = "audio"))]
//! # fn main() {}
//! ```
//!
//! ### Multimodal Tokenization
//!
//! ```rust,no_run
//! # #[cfg(all(feature = "audio", feature = "image"))]
//! # fn main() -> Result<(), Box<dyn std::error::Error>> {
//! use tekken::{Tekkenizer, Audio, Image, SpecialTokenPolicy};
//!
//! let tokenizer = Tekkenizer::from_file("tekken.json")?;
//!
//! // Text tokens
//! let text_tokens = tokenizer.encode("Please transcribe this audio:", true, false)?;
//!
//! // Image tokens (if tokenizer has image support)
//! if tokenizer.has_image_support() {
//! let image = Image::from_file("picture.png")?;
//! let image_encoding = tokenizer.encode_image(&image)?;
//! println!("Image tokens: {}", image_encoding.tokens.len());
//! }
//!
//! // Audio tokens (if tokenizer has audio support)
//! if tokenizer.has_audio_support() {
//! let audio = Audio::from_file("speech.wav")?;
//! let audio_encoding = tokenizer.encode_audio(audio)?;
//!
//! // Combine text and audio tokens
//! let mut combined_tokens = text_tokens;
//! combined_tokens.extend(audio_encoding.tokens);
//!
//! println!("Combined sequence: {} tokens", combined_tokens.len());
//! }
//! # Ok(())
//! # }
//! # #[cfg(not(all(feature = "audio", feature = "image")))]
//! # fn main() {}
//! ```
//!
//! ## Architecture
//!
//! The library is organized into several modules:
//!
//! - [`tekkenizer`]: Main tokenizer implementation and text processing
//! - [`image`]: Image loading, preprocessing, and image tokenization
//! - [`audio`]: Audio processing, mel-scale spectrograms, and audio tokenization
//! - [`special_tokens`]: Special token definitions and handling policies
//! - [`config`]: Configuration structures and version management
//! - [`model_settings`]: Model settings constraints for v15+ tokenizers
//! - [`errors`]: Comprehensive error handling
//!
//! ## Compatibility
//!
//! This Rust implementation is designed to be fully compatible with Mistral's Python
//! tokenizer implementation:
//!
//! - Identical tokenization results for text
//! - Same image resizing, normalization and token layout
//! - Same audio processing pipeline and token generation
//! - Compatible special token handling
//! - Matching mel filter bank computations
//!
//! ## Performance
//!
//! The Rust implementation provides significant performance improvements over Python:
//!
//! - Memory-safe processing with zero-copy operations where possible
//! - Efficient audio processing with optimized mel-scale computations
//! - Fast BPE tokenization using proven algorithms
//! - Minimal allocations and efficient data structures
// Re-export commonly used types for convenience
pub use ;
pub use ;
pub use ;
pub use ;
pub use ;
pub use ;
pub use ;
pub use SpecialTokenInfo;
pub use ;
pub use Tekkenizer;