Struct TokenizeOutput 

pub struct TokenizeOutput {
    pub tokenizer_mode: String,
    pub normalized_text: String,
    pub tokens: Vec<String>,
    pub fts_query: String,
}

分词输出（中英双语）。 Tokenization output payload (bilingual).

Fields

tokenizer_mode: String

实际使用的分词模式。 Effective tokenizer mode used by the engine.

normalized_text: String

轻量规范化后的原文。 Lightly normalized text.

tokens: Vec<String>

切分后的词元列表。 Token list after segmentation.

fts_query: String

面向 SQLite FTS 查询的安全表达式。 Safe SQLite FTS query expression derived from the tokens.

Trait Implementations

impl Clone for TokenizeOutput

fn clone(&self) -> TokenizeOutput

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenizeOutput

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for TokenizeOutput

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl PartialEq for TokenizeOutput

fn eq(&self, other: &TokenizeOutput) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Serialize for TokenizeOutput

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl Eq for TokenizeOutput

impl StructuralPartialEq for TokenizeOutput