Struct SimpleTokenizer 

pub struct SimpleTokenizer { /* private fields */ }

适用于拼音和中文的分词器

Implementations

impl SimpleTokenizer

pub fn disable_pinyin(&mut self)

关闭拼音分词

pub fn disable_stopword(&mut self)

不启用停词表

pub fn tokenize_query(text: &str) -> Option<String>

将查询文档转换成 SQLite 的 match 语句

Trait Implementations

impl Default for SimpleTokenizer

fn default() -> Self

Returns the “default value” for a type.

impl Tokenizer for SimpleTokenizer

type Global = ()

一个全局数据的类型

fn name() -> &'static CStr

提供一个 tokenizer 名称

fn new(_global: &Self::Global, args: Vec<String>) -> Result<Self, Error>

创建 Tokenizer 方法

fn tokenize<TKF>( &mut self, _reason: TokenizeReason, text: &[u8], push_token: TKF, ) -> Result<(), Error>

where TKF: FnMut(&[u8], Range<usize>, bool) -> Result<(), Error>,

分词的具体实现