Struct Token 

pub struct Token {
    pub text: String,
    pub byte_start: usize,
    pub byte_end: usize,
    pub position: usize,
    pub position_length: usize,
    pub word_id: WordId,
    pub details: Vec<String>,
}

Fields

text: String

Text content of the token.

byte_start: usize

Starting position of the token in bytes.

byte_end: usize

Ending position of the token in bytes.

position: usize

Position, expressed in number of tokens.

position_length: usize

The length expressed in terms of number of original tokens.

word_id: WordId

The ID of the word and a flag to indicate whether the word is registered in the dictionary.

details: Vec<String>

Detailes about the token. It contains metadata for tokens, such as part-of-speech information.

Trait Implementations

impl Clone for Token

fn clone(&self) -> Token

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Serialize for Token

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.