TokenFunction

lindera_sqlite

Type Alias TokenFunction 

Source 
pub type TokenFunction = extern "C" fn(p_ctx: *mut c_void, t_flags: c_int, p_token: *const c_char, n_token: c_int, i_start: c_int, i_end: c_int) -> c_int;
Expand description

Token callback function type.

This type represents the callback function provided by SQLite FTS5 for each token produced during tokenization. The extension calls this function once per token.

§Parameters

  • p_ctx - Context pointer passed through from the tokenization call
  • t_flags - Token flags (currently always 0 in this implementation)
  • p_token - Pointer to the token text (UTF-8 encoded)
  • n_token - Length of the token in bytes
  • i_start - Byte offset where the token starts in the original text
  • i_end - Byte offset where the token ends in the original text

§Returns

  • SQLITE_OK - Token processed successfully, continue tokenization
  • Other codes - Error occurred, stop tokenization

§Example Flow

Input: "日本語" (9 bytes in UTF-8)

Callback 1: token="日本", n_token=6, i_start=0, i_end=6
Callback 2: token="語",   n_token=3, i_start=6, i_end=9