Struct ParallelRegexBPETokenizerConfig 

pub struct ParallelRegexBPETokenizerConfig { /* private fields */ }

Configuration for training a regex-based BPE tokenizer in parallel with rayon.

This struct implements Trainable and produces a ParallelRegexBPETokenizer.

Implementations

impl ParallelRegexBPETokenizerConfig

pub fn build( vocab_size: u32, pattern: Option<&str>, ) -> Result<Self, RegexBPETokenizerConfigError>

Create a new configuration for training a regex BPE tokenizer.

Arguments

• vocab_size - The desired vocabulary size (must be at least 256).
• pattern - Optional custom regex pattern. If None, uses the GPT-4 split pattern.

Returns

• Ok(RegexBPETokenizerConfig) if configuration is valid.
• Err(RegexBPETokenizerConfigError) if vocab size is too small or pattern is invalid.

pub fn from_merges( merges: u32, pattern: Option<&str>, ) -> Result<Self, RegexBPETokenizerConfigError>

Create a new configuration from the number of merges instead of vocab size.

Arguments

• merges - The number of merge operations to perform.
• pattern - Optional custom regex pattern.

Trait Implementations

impl Debug for ParallelRegexBPETokenizerConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Deserializable for ParallelRegexBPETokenizerConfig

fn load(&self, path: &Path) -> Result<Self::Output, Error>

Loads a ParallelRegexBPETokenizer from a file.

The file must contain a header line with the regex pattern, followed by merge rules (one per line).

Arguments

• path - The path to load the tokenizer from. Must have a .stok extension.

Returns

• Ok(ParallelRegexBPETokenizer) if the tokenizer was loaded successfully.
• Err(std::io::Error) if the file extension is invalid, reading fails, or the file format is invalid.

type Output = ParallelRegexBPETokenizer

The tokenizer type produced by loading.

impl Trainable for ParallelRegexBPETokenizerConfig

type Output = ParallelRegexBPETokenizer

The tokenizer type produced by training.

type TrainingError = Infallible

Error that could happen during training.

fn train( &self, dataset: &str, ) -> Result<ParallelRegexBPETokenizer, Self::TrainingError>

Trains a tokenizer on a given dataset to a target vocabulary size.