Trait rust_tokenizers::ConsolidatableTokens [−][src]
pub trait ConsolidatableTokens<T> where
T: TokenTrait, { fn iter_consolidate_tokens(&self) -> ConsolidatedTokenIterator<'_, T>ⓘ; }Notable traits for ConsolidatedTokenIterator<'a, T>
impl<'a, T> Iterator for ConsolidatedTokenIterator<'a, T> where
T: TokenTrait, type Item = &'a [T];
Expand description
ConsolidatableTokens
This trait can be implemented for collections of tokens (i.e. things that implement TokenTrait)
and instantiates an iterator to quickly iterate over the tokens in consolidated form, e.g.
grouping subtokens into words.
use rust_tokenizers::{ConsolidatableTokens, Token}; let tokens: Vec<Token> = vec![]; //add some tokens for (wordcount, word_tokens) in tokens.iter_consolidate_tokens().enumerate() { eprintln!("word #{} - {:?}", wordcount + 1, word_tokens); }
Required methods
fn iter_consolidate_tokens(&self) -> ConsolidatedTokenIterator<'_, T>ⓘNotable traits for ConsolidatedTokenIterator<'a, T>
impl<'a, T> Iterator for ConsolidatedTokenIterator<'a, T> where
T: TokenTrait, type Item = &'a [T];[src]
fn iter_consolidate_tokens(&self) -> ConsolidatedTokenIterator<'_, T>ⓘNotable traits for ConsolidatedTokenIterator<'a, T>
impl<'a, T> Iterator for ConsolidatedTokenIterator<'a, T> where
T: TokenTrait, type Item = &'a [T];[src]Creates an iterator from a sequence of ConsolidatableTokens.
Implementations on Foreign Types
fn iter_consolidate_tokens(&self) -> ConsolidatedTokenIterator<'_, Token>ⓘNotable traits for ConsolidatedTokenIterator<'a, T>
impl<'a, T> Iterator for ConsolidatedTokenIterator<'a, T> where
T: TokenTrait, type Item = &'a [T];[src]fn iter_consolidate_tokens(&self) -> ConsolidatedTokenIterator<'_, TokenRef<'a>>ⓘNotable traits for ConsolidatedTokenIterator<'a, T>
impl<'a, T> Iterator for ConsolidatedTokenIterator<'a, T> where
T: TokenTrait, type Item = &'a [T];[src]