Struct mako::tokenization::hf_tokenizers::tokenizer::normalizer::NormalizedString

pub struct NormalizedString { /* private fields */ }

A NormalizedString takes care of processing an “original” string to modify it and obtain a “normalized” string. It keeps both version of the string, alignments information between both and provides an interface to retrieve ranges of each string, using offsets from any of them.

It is possible to retrieve a part of the original string, by indexing it with offsets from the normalized one, and the other way around too. It is also possible to convert offsets from one referential to the other one easily.

Implementations

impl NormalizedString

pub fn get(&self) -> &str

Return the normalized string

pub fn get_original(&self) -> &str

Return the original string

pub fn offsets_original(&self) -> Offsets

Return the original offsets

pub fn convert_offsets<T>(&self, range: Range<T>) -> Option<Range<usize>> where
    T: RangeBounds<usize> + Clone, 

Convert the given offsets range from one referential to the other one: Original => Normalized or Normalized => Original

Returns None when targeting something that is outside range

pub fn get_range<T>(&self, range: Range<T>) -> Option<&str> where
    T: RangeBounds<usize> + Clone, 

Return a range of the normalized string

pub fn get_range_original<T>(&self, range: Range<T>) -> Option<&str> where
    T: RangeBounds<usize> + Clone, 

Return a range of the original string

pub fn slice<T>(&self, range: Range<T>) -> Option<NormalizedString> where
    T: RangeBounds<usize> + Clone, 

Return a slice of the current NormalizedString If the range is not on char boundaries, return None

pub fn transform_range<T, I>(
    &mut self,
    range: Range<T>,
    dest: I,
    initial_offset: usize
) where
    T: RangeBounds<usize> + Clone,
    I: IntoIterator<Item = (char, isize)>, 

Applies transformations to the current normalized version of the string, while updating the alignments. This method expect an Iterator yielding each char of the new normalized string with a change isize equals to:

• 1 if this is a new char
• -N if the char is right before N removed chars
• 0 if the char is replacing the existing one Since it is possible that the normalized string doesn’t include some of the characters at the beginning of the original one, we need an initial_offset which represents the number of removed chars at the very beginning.

pub fn transform<I>(&mut self, dest: I, initial_offset: usize) where
    I: IntoIterator<Item = (char, isize)>, 

Applies transformations to the current normalized version of the string, while updating the alignments. This method expect an Iterator yielding each char of the new normalized string with a change isize equals to:

• 1 if this is a new char
• -N if the char is right before N removed chars
• 0 if the char is replacing the existing one Since it is possible that the normalized string doesn’t include some of the characters at the beginning of the original one, we need an initial_offset which represents the number of removed chars at the very beginning.

pub fn nfd(&mut self) -> &mut Self

Applies NFD normalization

pub fn nfkd(&mut self) -> &mut Self

Applies NFKD normalization

pub fn nfc(&mut self) -> &mut Self

Applies NFC normalization

pub fn nfkc(&mut self) -> &mut Self

Applies NFKC normalization

pub fn filter<F: Fn(char) -> bool>(&mut self, keep: F) -> &mut Self

Applies filtering over our characters

pub fn prepend(&mut self, s: &str) -> &mut Self

Prepend the given string to ourself

pub fn append(&mut self, s: &str) -> &mut Self

Append the given string to ourself

pub fn map<F: Fn(char) -> char>(&mut self, map: F) -> &mut Self

Map our characters

pub fn for_each<F: FnMut(char)>(&self, foreach: F) -> &Self

Calls the given function for each characters

pub fn lowercase(&mut self) -> &mut Self

Lowercase

pub fn uppercase(&mut self) -> &mut Self

Uppercase

pub fn replace<P: Pattern>(&mut self, pattern: P, content: &str) -> Result<()>

Replace anything that matches the pattern with the given content.

pub fn clear(&mut self) -> usize

Clear the normalized part of the string

pub fn split<P: Pattern>(
    &self,
    pattern: P,
    behavior: SplitDelimiterBehavior
) -> Result<Vec<NormalizedString>>

Split the current string in many subparts. Specify what to do with the delimiter.

Splitting Behavior for the delimiter

The behavior can be one of the followings: When splitting on '-' for example, with input the-final--countdown:

• Removed => [ "the", "", "final", "", "", "countdown" ]
• Isolated => [ "the", "-", "final", "-", "-", "countdown" ]
• MergedWithPrevious => [ "the-", "final-", "-", "countdown" ]
• MergedWithNext => [ "the", "-final", "-", "-countdown" ]

pub fn lstrip(&mut self) -> &mut Self

Remove any leading space(s) of the normalized string

pub fn rstrip(&mut self) -> &mut Self

Remove any trailing space(s) of the normalized string

pub fn strip(&mut self) -> &mut Self

Remove any leading and trailing space(s) of the normalized string

pub fn len(&self) -> usize

Returns the length of the normalized string (counting chars not bytes)

pub fn len_original(&self) -> usize

Returns the length of the original string (counting chars not bytes)

pub fn is_empty(&self) -> bool

Whether empty

Trait Implementations

impl Clone for NormalizedString

fn clone(&self) -> NormalizedString

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for NormalizedString

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for NormalizedString

fn default() -> NormalizedString

Returns the “default value” for a type.

impl From<&'_ str> for NormalizedString

fn from(s: &str) -> Self

Performs the conversion.

impl From<NormalizedString> for Split

fn from(n: NormalizedString) -> Self

Performs the conversion.

impl From<NormalizedString> for PreTokenizedString

fn from(s: NormalizedString) -> Self

Performs the conversion.

impl From<String> for NormalizedString

fn from(s: String) -> Self

Performs the conversion.

impl PartialEq<NormalizedString> for NormalizedString

fn eq(&self, other: &NormalizedString) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &NormalizedString) -> bool

This method tests for !=.

impl StructuralPartialEq for NormalizedString