Trait rust_tokenizers::TokenTrait

source · 
pub trait TokenTrait {
    // Required methods
    fn offset(&self) -> Option<Offset>;
    fn mask(&self) -> Mask;
    fn as_str(&self) -> &str;
}
Expand description

Token abstraction trait to access token fields, irrespective of their form (reference of owned)

Required Methods§

source

fn offset(&self) -> Option<Offset>

Returns the offset of the token with respect to the original string

source

fn mask(&self) -> Mask

Returns the token mask

source

fn as_str(&self) -> &str

Returns a string representation for the token

Implementors§