Struct notmecab::LexerToken

pub struct LexerToken {
    pub cost: i64,
    pub real_cost: i64,
    pub start: usize,
    pub end: usize,
    pub kind: TokenType,
    pub original_id: u32,
    pub feature_offset: u32,
    // some fields omitted
}

Fields

cost: i64

Used internally during lattice pathfinding.

real_cost: i64

Cost updated to include right-edge connection cost after parsing.

start: usize

Location, in bytes, of the surface of this LexerToken in the string it was parsed from.

end: usize

Corresponding ending location, in bytes. Exclusive. (i.e. when start+1 == end, the LexerToken's surface is one byte long)

kind: TokenType

Origin of token. BOS and UNK are virtual origins ("beginning/ending-of-string" and "unknown", respectively). Normal means it came from the mecab dictionary.

The BOS (beginning/ending-of-string) tokens are stripped away in parse_to_lexertokens.

original_id: u32

Unique identifier of what specific lexeme realization this is, from the mecab dictionary. changes between dictionary versions.

feature_offset: u32

Trait Implementations

impl Clone for LexerToken

fn clone(&self) -> LexerToken

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for LexerToken

fn fmt(&self, f: &mut Formatter) -> Result

Formats the value using the given formatter.