Struct BPETokenizer 

pub struct BPETokenizer { /* private fields */ }

Pure-Rust BPE encoder.

Construct via BPETokenizer::new; check BPETokenizer::supports first if you don’t know whether the map has the data BPE needs (use crate::Tokenize::pick which falls back).

Implementations

impl BPETokenizer

pub fn supports(map: &TokenizerMap) -> bool

Returns true if the map has the data BPETokenizer needs.

pub fn new(map: &TokenizerMap) -> Result<Self, String>

Construct a BPETokenizer from a map. Returns an error message if the map lacks the required vocab/merges/encoder.

pub fn encode(&self, text: &str) -> Vec<u32>

Encode text → token IDs.

Trait Implementations

impl ITokenizer for BPETokenizer

fn id(&self) -> &str

Identifier of the underlying vocabulary.

fn encode(&self, text: &str) -> Vec<u32>

Encode a string to a sequence of token IDs.