Struct notmecab::LexerToken

pub struct LexerToken {
    pub cost: i64,
    pub real_cost: i64,
    pub range: Range<usize>,
    pub kind: TokenType,
    pub original_id: u32,
    pub feature_offset: u32,
    // some fields omitted
}

Fields

cost: i64

Used internally during lattice pathfinding.

real_cost: i64

Cost updated to include right-edge connection cost after parsing.

range: Range<usize>

The range, in bytes, to which this token corresponds to in the original text.

kind: TokenType

Origin of token. BOS and UNK are virtual origins ("beginning/ending-of-string" and "unknown", respectively). Normal means it came from the mecab dictionary.

The BOS (beginning/ending-of-string) tokens are stripped away in parse_to_lexertokens.

original_id: u32

Unique identifier of what specific lexeme realization this is, from the mecab dictionary. changes between dictionary versions.

feature_offset: u32

Implementations

impl LexerToken

pub fn get_text<'a>(&self, whole_text: &'a str) -> &'a str

Returns the text to which this token corresponds to in the original text.

The whole_text is the original string for which you've called Dict::tokenize or Dict::tokenize_with_cache.

pub fn get_feature<'a>(&self, dict: &'a Dict) -> &'a str

Returns a feature string corresponding to this token.

Feature strings are dictionary-specific so unfortunately you need to parse them yourself. They usually contain things like the exact part-of-speech this token represents, its reading, whenever it's conjugated or not, etc.

Trait Implementations

impl Clone for LexerToken

fn clone(&self) -> LexerToken

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for LexerToken

fn fmt(&self, f: &mut Formatter) -> Result

Formats the value using the given formatter.