Trait rust_tokenizers::ConsolidatableTokens

pub trait ConsolidatableTokens<T>where
    T: TokenTrait,{
    // Required method
    fn iter_consolidate_tokens(&self) -> ConsolidatedTokenIterator<'_, T> ;
}

ConsolidatableTokens

This trait can be implemented for collections of tokens (i.e. things that implement TokenTrait) and instantiates an iterator to quickly iterate over the tokens in consolidated form, e.g. grouping subtokens into words.

use rust_tokenizers::{ConsolidatableTokens, Token};
let tokens: Vec<Token> = vec![]; //add some tokens
for (wordcount, word_tokens) in tokens.iter_consolidate_tokens().enumerate() {
    eprintln!("word #{} - {:?}", wordcount + 1, word_tokens);
}

Required Methods

fn iter_consolidate_tokens(&self) -> ConsolidatedTokenIterator<'_, T>

Creates an iterator from a sequence of ConsolidatableTokens.

Implementations on Foreign Types

impl ConsolidatableTokens<Token> for Vec<Token>

fn iter_consolidate_tokens(&self) -> ConsolidatedTokenIterator<'_, Token>

impl<'a> ConsolidatableTokens<TokenRef<'a>> for Vec<TokenRef<'a>>

fn iter_consolidate_tokens(&self) -> ConsolidatedTokenIterator<'_, TokenRef<'a>>

Implementors