Struct tokenizers::tokenizer::NormalizedString

pub struct NormalizedString { /* fields omitted */ }

A NormalizedString takes care of processing an "original" string to modify it and obtain a "normalized" string. It keeps both version of the string, alignments information between both and provides an interface to retrieve ranges of each string, using offsets from any of them.

It is possible to retrieve a part of the original string, by indexing it with offsets from the normalized one, and the other way around too. It is also possible to convert offsets from one referential to the other one easily.

Methods

impl NormalizedString

pub fn from(s: &str) -> Self

Create a NormalizedString from the given str

pub fn get(&self) -> &str

Return the normalized string

pub fn get_original(&self) -> &str

Return the original string

pub fn convert_offsets<T: RangeBounds<usize>>(
    &self,
    range: Range<T>
) -> Option<Range<usize>>

Convert the given offsets range from one referential to the other one: Original => Normalized or Normalized => Original

pub fn get_range<T: RangeBounds<usize>>(&self, range: Range<T>) -> Option<&str>

Return a range of the normalized string (indexing on char not bytes)

pub fn get_range_original<T: RangeBounds<usize>>(
    &self,
    range: Range<T>
) -> Option<&str>

Return a range of the original string (indexing on char not bytes)

pub fn transform<I: Iterator<Item = (char, isize)>>(
    &mut self,
    dest: I,
    initial_offset: usize
)

Applies transformations to the current normalized version, updating the current alignments with the new ones. This method expect an Iterator yielding each char of the new normalized string with a change isize equals to:

• 1 if this is a new char
• -N if the char is right before N removed chars
• 0 if this char represents the old one (even if changed) Since it is possible that the normalized string doesn't include some of the characters at the beginning of the original one, we need an initial_offset which represents the number of removed chars at the very beginning.

change should never be more than 1. If multiple chars are added, each of them has a change of 1, but more doesn't make any sense. We treat any value above 1 as 1.

pub fn nfd(&mut self) -> &mut Self

Applies NFD normalization

pub fn nfkd(&mut self) -> &mut Self

Applies NFKD normalization

pub fn nfc(&mut self) -> &mut Self

Applies NFC normalization

pub fn nfkc(&mut self) -> &mut Self

Applies NFKC normalization

pub fn filter<F: Fn(&char) -> bool>(&mut self, filter: F) -> &mut Self

Applies filtering over our characters

pub fn prepend(&mut self, s: &str) -> &mut Self

Prepend the given string to ourself

pub fn append(&mut self, s: &str) -> &mut Self

Append the given string to ourself

pub fn map<F: Fn(char) -> char>(&mut self, map: F) -> &mut Self

Map our characters

pub fn for_each<F: FnMut(char)>(&mut self, foreach: F) -> &mut Self

Calls the given function for each characters

pub fn lowercase(&mut self) -> &mut Self

Lowercase

pub fn uppercase(&mut self) -> &mut Self

Uppercase

pub fn split_off(&mut self, at: usize) -> Self

Split off ourselves, returning a new Self that contains the range [at, len). self will then contain the range [0, at). The provided at indexes on char not bytes.

pub fn merge_with(&mut self, other: &NormalizedString)

Merge with the given NormalizedString by appending it to self

pub fn lstrip(&mut self) -> &mut Self

Remove any leading space(s) of the normalized string

pub fn rstrip(&mut self) -> &mut Self

Remove any trailing space(s) of the normalized string

pub fn strip(&mut self) -> &mut Self

Remove any leading and trailing space(s) of the normalized string

pub fn len(&self) -> usize

Returns the length of the normalized string (counting chars not bytes)

pub fn len_original(&self) -> usize

Returns the length of the original string (counting chars not bytes)

pub fn is_empty(&self) -> bool

Whether empty

Trait Implementations

impl Clone for NormalizedString

fn clone(&self) -> NormalizedString

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for NormalizedString

fn fmt(&self, f: &mut Formatter) -> Result

Formats the value using the given formatter.

impl Default for NormalizedString

fn default() -> NormalizedString

Returns the "default value" for a type.

impl PartialEq<NormalizedString> for NormalizedString

fn eq(&self, other: &NormalizedString) -> bool

This method tests for self and other values to be equal, and is used by ==.

#[must_use]fn ne(&self, other: &Rhs) -> bool

This method tests for !=.