Struct FtsTokenizer 

pub struct FtsTokenizer { /* private fields */ }

Full-text search tokenizer for Thai text.

Wraps Tokenizer with stopword filtering, synonym expansion, and n-gram generation for out-of-vocabulary tokens.

Construct once and reuse:

use kham_core::fts::FtsTokenizer;

let fts = FtsTokenizer::new();
let tokens = fts.segment_for_fts("กินข้าวกับปลา");
assert!(!tokens.is_empty());

Implementations

impl FtsTokenizer

pub fn new() -> Self

Create an FtsTokenizer with built-in stopwords and no synonyms.

pub fn builder() -> FtsTokenizerBuilder

Return a FtsTokenizerBuilder for custom configuration.

pub fn segment_for_fts(&self, text: &str) -> Vec<FtsToken>

Segment text and annotate each token for FTS indexing.

Normalises the input text before segmentation so that สระลอย and stacked tone marks are handled correctly. Whitespace tokens are excluded.

The returned Vec<FtsToken> covers all non-whitespace tokens. Call index_tokens instead when you only need the tokens to be indexed (stopwords excluded).

pub fn index_tokens(&self, text: &str) -> Vec<FtsToken>

Return only the tokens to be written into a search index.

Filters out stopwords and whitespace. Each FtsToken still carries its original position so phrase-distance scoring remains correct.

pub fn lexemes(&self, text: &str) -> Vec<String>

Collect all lexeme strings to be stored in a tsvector.

Returns one string per non-stop token, plus synonym expansions and trigrams for unknown tokens. Duplicates are not removed (the caller or PostgreSQL handles deduplication).

Trait Implementations

impl Default for FtsTokenizer

fn default() -> Self

Returns the “default value” for a type.