Struct Tokenizer 

pub struct Tokenizer {
    pub segmenter: Segmenter,
    pub character_filters: Vec<BoxCharacterFilter>,
    pub token_filters: Vec<BoxTokenFilter>,
}

Fields

segmenter: Segmenter

Segmenter The segmenter field is an instance of the Segmenter struct, which is responsible for segmenting text into tokens. This is a core component of the tokenizer, enabling it to break down input text into manageable and meaningful units for further processing.

character_filters: Vec<BoxCharacterFilter>

Character filters A vector of boxed character filters that will be applied to the input text before tokenization. Each character filter is responsible for transforming the input text in a specific way, such as normalizing characters or removing unwanted characters.

token_filters: Vec<BoxTokenFilter>

Token filters A vector of boxed token filters that will be applied to the tokens during tokenization. Each token filter is a boxed trait object implementing the TokenFilter trait, allowing for various transformations and processing steps to be applied to the tokens.

Implementations

impl Tokenizer

pub fn new(segmenter: Segmenter) -> Self

Creates a new Tokenizer instance from a provided Segmenter.

Arguments

• segmenter - An instance of the Segmenter struct, which is responsible for the core tokenization process.

Returns

Returns a new Tokenizer instance that uses the provided segmenter for tokenization, with empty character and token filters.

Details

• segmenter: The segmenter is responsible for handling the actual segmentation and tokenization of text. It is passed into the Tokenizer during initialization.
• character_filters: This is initialized as an empty vector and can be modified later to include character filters.
• token_filters: This is also initialized as an empty vector and can be modified later to include token filters.

pub fn from_config(config: &TokenizerConfig) -> LinderaResult<Self>

pub fn append_character_filter( &mut self, character_filter: BoxCharacterFilter, ) -> &mut Self

Appends a character filter to the tokenizer.

Arguments

• character_filter - A BoxCharacterFilter that will be added to the tokenizer. This filter will be applied to the text during the tokenization process.

Returns

Returns a mutable reference to Self, allowing for method chaining.

Details

• This method adds a new character filter to the Tokenizer’s character_filters vector.
• It returns a mutable reference to self, allowing multiple character filters to be appended in a chain of method calls.

pub fn append_token_filter(&mut self, token_filter: BoxTokenFilter) -> &mut Self

Appends a token filter to the tokenizer.

Arguments

• token_filter - A BoxTokenFilter that will be added to the tokenizer. This filter will be applied to the tokens after they are segmented.

Returns

Returns a mutable reference to Self, allowing for method chaining.

Details

• This method adds a new token filter to the Tokenizer’s token_filters vector.
• It returns a mutable reference to self, allowing multiple token filters to be appended in a chain of method calls.

pub fn tokenize<'a>(&'a self, text: &'a str) -> LinderaResult<Vec<Token<'a>>>

Tokenizes the input text using the tokenizer’s segmenter, character filters, and token filters.

Arguments

• text - A reference to the input text (&str) that will be tokenized.

Returns

Returns a LinderaResult containing a vector of Tokens, where each Token represents a segment of the tokenized text.

Process

1. Apply character filters:
   • If any character filters are defined, they are applied to the input text before tokenization.
   • The offsets, diffs, and text_len are recorded for each character filter.
2. Segment the text:
   • The segmenter divides the (potentially filtered) text into tokens.
3. Apply token filters:
   • If any token filters are defined, they are applied to the segmented tokens.
4. Correct token offsets:
   • If character filters were applied, the byte offsets of each token are corrected to account for changes introduced by those filters.

Errors

• Returns an error if any of the character or token filters fail during processing.
• Returns an error if the segmentation process fails.

Details

• Cow<'a, str> is used for the normalized_text, allowing the function to either borrow the original text or create an owned version if the text needs modification.
• If no character filters are applied, the original text is used as-is for segmentation.
• Token offsets are adjusted after the tokenization process if character filters were applied to ensure the byte positions of each token are accurate relative to the original text.

pub fn tokenize_with_lattice<'a>( &'a self, text: &'a str, lattice: &mut Lattice, ) -> LinderaResult<Vec<Token<'a>>>

Tokenizes the input text using the tokenizer’s segmenter, character filters, and token filters.

Arguments

• text - A reference to the input text (&str) that will be tokenized.
• lattice - A mutable reference to a Lattice structure. This allows reusing the lattice across multiple calls to avoid memory allocation.

Returns

Returns a LinderaResult containing a vector of Tokens, where each Token represents a segment of the tokenized text.

Process

1. Apply character filters:
   • If any character filters are defined, they are applied to the input text before tokenization.
   • The offsets, diffs, and text_len are recorded for each character filter.
2. Segment the text:
   • The segmenter divides the (potentially filtered) text into tokens.
3. Apply token filters:
   • If any token filters are defined, they are applied to the segmented tokens.
4. Correct token offsets:
   • If character filters were applied, the byte offsets of each token are corrected to account for changes introduced by those filters.

Errors

• Returns an error if any of the character or token filters fail during processing.
• Returns an error if the segmentation process fails.

Details

• Cow<'a, str> is used for the normalized_text, allowing the function to either borrow the original text or create an owned version if the text needs modification.
• If no character filters are applied, the original text is used as-is for segmentation.
• Token offsets are adjusted after the tokenization process if character filters were applied to ensure the byte positions of each token are accurate relative to the original text.

pub fn tokenize_nbest<'a>( &'a self, text: &'a str, n: usize, unique: bool, cost_threshold: Option<i64>, ) -> LinderaResult<Vec<(Vec<Token<'a>>, i64)>>

Tokenizes the input text and returns the top-N results.

Each result is a Vec<Token> with character/token filters applied. Results are ordered by cost (best first).

pub fn tokenize_nbest_with_lattice<'a>( &'a self, text: &'a str, lattice: &mut Lattice, n: usize, unique: bool, cost_threshold: Option<i64>, ) -> LinderaResult<Vec<(Vec<Token<'a>>, i64)>>

Tokenizes the input text and returns the top-N results with costs. Each result is a (tokens, cost) pair. If unique is true, results with the same word boundaries are deduplicated. If cost_threshold is Some(t), paths whose cost exceeds best_cost + t are discarded.

Trait Implementations

impl Clone for Tokenizer

fn clone(&self) -> Self

Creates a deep clone of the Tokenizer instance, including all character filters, token filters, and the segmenter.

Returns

Returns a new Tokenizer instance that is a deep clone of the current instance. All internal filters and the segmenter are cloned.

Details

• Character Filters: Each character filter is cloned by calling its box_clone method, which ensures that any dynamically dispatched filters are properly cloned.
• Token Filters: Similarly, each token filter is cloned using the box_clone method to handle dynamic dispatch.
• Segmenter: The segmenter is cloned using its clone method.

Notes

• This method performs deep cloning, meaning that all internal filters and segmenter instances are fully duplicated.
• The box_clone method is used to clone the dynamically dispatched filter objects (BoxCharacterFilter and BoxTokenFilter).

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.