Struct tantivy_jieba::JiebaTokenizer [−][src]
pub struct JiebaTokenizer;
Expand description
Tokenize the text using jieba_rs.
Need to load dict on first tokenization.
Example
use tantivy::tokenizer::*; let tokenizer = tantivy_jieba::JiebaTokenizer {}; let mut token_stream = tokenizer.token_stream("测试"); assert_eq!(token_stream.next().unwrap().text, "测试"); assert!(token_stream.next().is_none());
Register tantivy tokenizer
use tantivy::schema::Schema; use tantivy::tokenizer::*; use tantivy::Index; let tokenizer = tantivy_jieba::JiebaTokenizer {}; let index = Index::create_in_ram(schema); index.tokenizers() .register("jieba", tokenizer);
Trait Implementations
Creates a token stream for a given str
.
Auto Trait Implementations
impl RefUnwindSafe for JiebaTokenizer
impl Send for JiebaTokenizer
impl Sync for JiebaTokenizer
impl Unpin for JiebaTokenizer
impl UnwindSafe for JiebaTokenizer
Blanket Implementations
Mutably borrows from an owned value. Read more
impl<T> Downcast for T where
T: Any,
impl<T> Downcast for T where
T: Any,
Convert Box<dyn Trait>
(where Trait: Downcast
) to Box<dyn Any>
. Box<dyn Any>
can
then be further downcast
into Box<ConcreteType>
where ConcreteType
implements Trait
. Read more
pub fn into_any_rc(self: Rc<T>) -> Rc<dyn Any + 'static>
pub fn into_any_rc(self: Rc<T>) -> Rc<dyn Any + 'static>
Convert Rc<Trait>
(where Trait: Downcast
) to Rc<Any>
. Rc<Any>
can then be
further downcast
into Rc<ConcreteType>
where ConcreteType
implements Trait
. Read more
Convert &Trait
(where Trait: Downcast
) to &Any
. This is needed since Rust cannot
generate &Any
’s vtable from &Trait
’s. Read more
pub fn as_any_mut(&mut self) -> &mut (dyn Any + 'static)
pub fn as_any_mut(&mut self) -> &mut (dyn Any + 'static)
Convert &mut Trait
(where Trait: Downcast
) to &Any
. This is needed since Rust cannot
generate &mut Any
’s vtable from &mut Trait
’s. Read more