bitcannon 0.1.0-dev.0

SIMD Unicode atom classification + bitstream pre-tokenization: one classify pass into byte-exact token spans (cl100k, o200k, GPT-2/ByteLevel, DeepSeek, Whitespace, Bert, ...).
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

//! Mistral `tekken` (mistral-small-4 / mistral-4). o200k's grammar with two changes: letter tokens
//! take no contraction suffix, and the digit rule is a bare `\p{N}` — one token per digit.
//!
//! The grammar lives in [`super::family_o200k`]; this is o200k's regex at `CONTR = false, DIGITS = 1`.

use crate::{AUX_SLASH, Span};

/// Mistral tekken pre-tokenization.
#[must_use]
pub fn bitcannon_tekken(
    text: &[u8],
    tags: &[u8],
    starts: &mut [u64],
    flag: &mut [u64],
    later: &mut [u64],
    out: &mut [Span],
) -> usize {
    super::family_o200k::run::<{ AUX_SLASH }, false, 1, false>(text, tags, starts, flag, later, out)
}