Trait tokenizers::tokenizer::PreTokenizer

pub trait PreTokenizer: Send + Sync {
    fn pre_tokenize(
        &self, 
        normalized: &mut NormalizedString
    ) -> Result<Vec<(String, Offsets)>>;
}

The PreTokenizer is in charge of doing the pre-segmentation step. It splits the given string in multiple substrings, keeping track of the offsets of said substrings from the NormalizedString. In some occasions, the PreTokenizer might need to modify the given NormalizedString to ensure we can entirely keep track of the offsets and the mapping with the original string.

Required methods

fn pre_tokenize(
    &self,
    normalized: &mut NormalizedString
) -> Result<Vec<(String, Offsets)>>

Loading content...

Implementors

impl PreTokenizer for BertPreTokenizer

fn pre_tokenize(
    &self,
    normalized: &mut NormalizedString
) -> Result<Vec<(String, Offsets)>>

impl PreTokenizer for ByteLevel

As a PreTokenizer, ByteLevel is in charge of transforming all the unicode characters into their byte-level counterpart. It also splits the input according to the configured regex.

fn pre_tokenize(
    &self,
    normalized: &mut NormalizedString
) -> Result<Vec<(String, Offsets)>>

impl PreTokenizer for CharDelimiterSplit

fn pre_tokenize(
    &self,
    normalized: &mut NormalizedString
) -> Result<Vec<(String, Offsets)>>

impl PreTokenizer for Metaspace

fn pre_tokenize(
    &self,
    normalized: &mut NormalizedString
) -> Result<Vec<(String, Offsets)>>

impl PreTokenizer for Whitespace

fn pre_tokenize(
    &self,
    normalized: &mut NormalizedString
) -> Result<Vec<(String, Offsets)>>

impl PreTokenizer for WhitespaceSplit

fn pre_tokenize(
    &self,
    normalized: &mut NormalizedString
) -> Result<Vec<(String, Offsets)>>

Loading content...