Struct FtsToken 

pub struct FtsToken {
    pub text: String,
    pub position: usize,
    pub kind: TokenKind,
    pub is_stop: bool,
    pub synonyms: Vec<String>,
    pub trigrams: Vec<String>,
    pub pos: Option<PosTag>,
    pub ne: Option<NamedEntityKind>,
    pub confidence: f32,
}

A token produced by the FTS pipeline, ready for lexeme indexing.

Fields

text: String

The token text (owned; may be normalised).

position: usize

Ordinal position in the token sequence (0-based, gaps for whitespace).

kind: TokenKind

Script / category of the original token.

is_stop: bool

true if this token matches the stopword list.

synonyms: Vec<String>

Synonym expansions (empty if none configured or no match).

trigrams: Vec<String>

Character trigrams — populated only for TokenKind::Unknown tokens.

pos: Option<PosTag>

Primary part-of-speech tag from the lookup table, or None if the word is not in the table (OOV) or is not a Thai token.

ne: Option<NamedEntityKind>

Named entity category, or None if the token is not in the NE gazetteer. When set, kind is TokenKind::Named(ne).

confidence: f32

Segmentation confidence in the range [0.0, 1.0]. 0.0 = Unknown token (no dictionary evidence). 1.0 = unambiguous high-frequency dictionary match.

Trait Implementations

impl Clone for FtsToken

fn clone(&self) -> FtsToken

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for FtsToken

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for FtsToken

fn eq(&self, other: &FtsToken) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for FtsToken