lance-tokenizer-12.0.0
Platform
- x86_64-unknown-linux-gnu
Feature flags

docs.rs

Rust

lance-tokenizer 12.0.0

Tokenizer abstractions and implementations for Lance

Crate
Source
Builds
Feature flags

..
stop_word_filter
alphanum_only.rs
analyzer.rs
ascii_folding_filter.rs
code_tokenizer.rs
icu.rs
jieba.rs
lib.rs
lindera.rs
lower_caser.rs
ngram_tokenizer.rs
raw_tokenizer.rs
remove_long.rs
simple_tokenizer.rs
stemmer.rs
stop_word_filter.rs
tokenizer_api.rs
whitespace_tokenizer.rs
word_delimiter_filter.rs