tekken/
lib.rs

1//! # Tekken - Rust Implementation of Mistral's Multimodal Tokenizer
2//!
3//! <div class="warning">
4//!
5//! **This crate has been renamed to [`tekken`](https://crates.io/crates/tekken).**
6//! Version 0.1.2 is the final `tekken-rs` release; all development continues in
7//! `tekken` starting with 0.2.0. The library name is unchanged, so migrating only
8//! requires replacing `tekken-rs` with `tekken = "0.2.0"` in your `Cargo.toml` —
9//! all `use tekken::...` paths keep working.
10//!
11//! </div>
12//!
13//! `tekken` is a Rust implementation of Mistral's Tekken tokenizer with full support
14//! for both text and audio tokenization. It provides high-performance, memory-safe
15//! tokenization that is fully compatible with the Python implementation.
16//!
17//! ## Features
18//!
19//! - **Text Tokenization**: Full BPE (Byte Pair Encoding) support with special tokens
20//! - **Audio Processing**: Convert audio waveforms to token sequences using mel-scale spectrograms
21//! - **Multimodal Support**: Mix text and audio tokens in a single sequence
22//! - **Version Compatibility**: Support for multiple tokenizer versions (V3, V7, V11, V13)
23//! - **Special Tokens**: Comprehensive handling of control, instruction, tool, and media tokens
24//!
25//! ## Quick Start
26//!
27//! ### Basic Text Tokenization
28//!
29//! ```rust,no_run
30//! use tekken::{Tekkenizer, SpecialTokenPolicy};
31//!
32//! # fn main() -> Result<(), Box<dyn std::error::Error>> {
33//! // Load tokenizer from configuration file
34//! let tokenizer = Tekkenizer::from_file("tekken.json")?;
35//!
36//! // Encode text with BOS/EOS tokens
37//! let text = "Hello, world!";
38//! let tokens = tokenizer.encode(text, true, true)?;
39//! println!("Tokens: {:?}", tokens);
40//!
41//! // Decode back to text
42//! let decoded = tokenizer.decode(&tokens, SpecialTokenPolicy::Keep)?;
43//! println!("Decoded: {}", decoded);
44//! # Ok(())
45//! # }
46//! ```
47//!
48//! ### Audio Tokenization
49//!
50//! ```rust,no_run
51//! use tekken::{Audio, AudioConfig, AudioSpectrogramConfig, AudioEncoder};
52//!
53//! # fn main() -> Result<(), Box<dyn std::error::Error>> {
54//! // Load audio file
55//! let audio = Audio::from_file("audio.wav")?;
56//!
57//! // Configure audio processing
58//! let spectrogram_config = AudioSpectrogramConfig::new(80, 160, 400)?;
59//! let audio_config = AudioConfig::new(16000, 12.5, spectrogram_config, None)?;
60//!
61//! // Create encoder and process audio
62//! let encoder = AudioEncoder::new(audio_config, 1000, 1001); // audio_token_id, begin_audio_token_id
63//! let encoding = encoder.encode(audio)?;
64//!
65//! println!("Audio encoded to {} tokens", encoding.tokens.len());
66//! # Ok(())
67//! # }
68//! ```
69//!
70//! ### Multimodal Tokenization
71//!
72//! ```rust,no_run
73//! use tekken::{Tekkenizer, Audio, SpecialTokenPolicy};
74//!
75//! # fn main() -> Result<(), Box<dyn std::error::Error>> {
76//! let tokenizer = Tekkenizer::from_file("tekken.json")?;
77//!
78//! // Text tokens
79//! let text_tokens = tokenizer.encode("Please transcribe this audio:", true, false)?;
80//!
81//! // Audio tokens (if tokenizer has audio support)
82//! if tokenizer.has_audio_support() {
83//!     let audio = Audio::from_file("speech.wav")?;
84//!     let audio_encoding = tokenizer.encode_audio(audio)?;
85//!     
86//!     // Combine text and audio tokens
87//!     let mut combined_tokens = text_tokens;
88//!     combined_tokens.extend(audio_encoding.tokens);
89//!     
90//!     println!("Combined sequence: {} tokens", combined_tokens.len());
91//! }
92//! # Ok(())
93//! # }
94//! ```
95//!
96//! ## Architecture
97//!
98//! The library is organized into several modules:
99//!
100//! - [`tekkenizer`]: Main tokenizer implementation and text processing
101//! - [`audio`]: Audio processing, mel-scale spectrograms, and audio tokenization  
102//! - [`special_tokens`]: Special token definitions and handling policies
103//! - [`config`]: Configuration structures and version management
104//! - [`errors`]: Comprehensive error handling
105//!
106//! ## Compatibility
107//!
108//! This Rust implementation is designed to be fully compatible with Mistral's Python
109//! tokenizer implementation:
110//!
111//! - Identical tokenization results for text
112//! - Same audio processing pipeline and token generation
113//! - Compatible special token handling
114//! - Matching mel filter bank computations
115//!
116//! ## Performance
117//!
118//! The Rust implementation provides significant performance improvements over Python:
119//!
120//! - Memory-safe processing with zero-copy operations where possible
121//! - Efficient audio processing with optimized mel-scale computations
122//! - Fast BPE tokenization using proven algorithms
123//! - Minimal allocations and efficient data structures
124
125pub mod audio;
126pub mod config;
127pub mod errors;
128pub mod special_tokens;
129pub mod tekkenizer;
130
131// Re-export commonly used types for convenience
132pub use audio::{Audio, AudioConfig, AudioEncoder, AudioSpectrogramConfig};
133pub use config::{TekkenConfig, TokenInfo};
134pub use errors::{Result, TokenizerError};
135pub use special_tokens::SpecialTokenInfo;
136pub use special_tokens::{SpecialTokenPolicy, SpecialTokens};
137pub use tekkenizer::Tekkenizer;
tekken/lib.rs

tekken/
lib.rs
