Struct SimpleBPETokenizerConfig 

pub struct SimpleBPETokenizerConfig { /* private fields */ }

Configuration for training a simple BPE tokenizer.

This struct implements Trainable and produces a SimpleBPETokenizer.

Implementations

impl SimpleBPETokenizerConfig

pub fn build(vocab_size: u32) -> Result<Self, VocabSizeTooSmall>

Create a new configuration for training a simple BPE tokenizer from vocab size.

pub fn from_merges(merges: u32) -> Self

Create a new configuration for training a simple BPE tokenizer from number of merges.

Trait Implementations

impl Debug for SimpleBPETokenizerConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Deserializable for SimpleBPETokenizerConfig

type Output = SimpleBPETokenizer

The tokenizer type produced by loading.

fn load(&self, path: &Path) -> Result<Self::Output, Error>

Loads a tokenizer from a file.

impl Trainable for SimpleBPETokenizerConfig

type Output = SimpleBPETokenizer

The tokenizer type produced by training.

type TrainingError = Infallible

Error that could happen during training.

fn train(&self, dataset: &str) -> Result<SimpleBPETokenizer, Infallible>

Trains a tokenizer on a given dataset to a target vocabulary size.