Struct Token 

pub struct Token {

    pub form: String,
    pub tag: String,
    pub position: usize,
    pub length: usize,
    pub word_position: usize,
    pub sent_position: usize,
    pub line_number: usize,
    pub sub_sent_position: usize,
    pub score: f32,
    pub typo_cost: f32,
    pub typo_form_id: u32,
    pub paired_token: Option<usize>,
    pub morpheme_id: Option<u32>,
    pub tag_id: Option<u8>,
    pub sense_or_script: Option<u8>,
    pub dialect: Option<u16>,
}

A single morpheme token produced by Kiwi analysis.

Fields

form: String

Surface form.

tag: String

Part-of-speech tag string.

position: usize

Character-based start offset in the original UTF-8 text (str.chars()).

length: usize

Character length (str.chars() count), not byte length.

word_position: usize

Word index inside the analyzed sentence.

sent_position: usize

Sentence index in multi-sentence analysis output.

line_number: usize

Line number metadata from Kiwi output.

sub_sent_position: usize

Sub-sentence index metadata from Kiwi output.

score: f32

Token score from language model.

typo_cost: f32

Typo correction cost for this token.

typo_form_id: u32

Typo form identifier from Kiwi internals.

paired_token: Option<usize>

Optional paired-token index (for paired punctuation etc.).

morpheme_id: Option<u32>

Optional morpheme id for dictionary-backed APIs.

tag_id: Option<u8>

Optional numeric tag id.

sense_or_script: Option<u8>

Optional sense id or script id depending on tag.

dialect: Option<u16>

Optional dialect id.

Trait Implementations

impl Clone for Token

fn clone(&self) -> Token

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.