Struct Split 

pub struct Split { /* private fields */ }

Split input strings using a pattern.

Implementations

impl Split

pub fn new(opts: SplitOptions<'_>) -> Result<Self, PreTokenizeError>

Construct a pre-tokenizer which splits input using a given regex pattern.

pub fn gpt2() -> Self

Split input strings into chunks using the GPT2_REGEX pattern originating from GPT-2 and subsequently used by many other models.

Use new to specify a custom pattern.

Trait Implementations

impl PreTokenizer for Split

fn pre_tokenize<'a>( &self, text: &'a str, ) -> Result<Vec<&'a str>, PreTokenizeError>

Split text into chunks and return a vector of sub-slices.