[−][src]Struct tantivy::tokenizer::TokenizerManager
The tokenizer manager serves as a store for all of the pre-configured tokenizer pipelines.
By default, it is populated with the following managers.
raw
: does not process nor tokenize the text.default
: Chops the text on according to whitespace and punctuation, removes tokens that are too long, and lowercases tokensen_stem
: Likedefault
, but also applies stemming on the resulting tokens. Stemming can improve the recall of your search engine.
Methods
impl TokenizerManager
[src]
pub fn register<A>(&self, tokenizer_name: &str, tokenizer: A) where
A: Into<BoxedTokenizer>,
[src]
A: Into<BoxedTokenizer>,
Registers a new tokenizer associated with a given name.
pub fn get(&self, tokenizer_name: &str) -> Option<BoxedTokenizer>
[src]
Accessing a tokenizer given its name.
Trait Implementations
impl Clone for TokenizerManager
[src]
fn clone(&self) -> TokenizerManager
[src]
fn clone_from(&mut self, source: &Self)
1.0.0[src]
impl Default for TokenizerManager
[src]
fn default() -> TokenizerManager
[src]
Creates an TokenizerManager
prepopulated with
the default pre-configured tokenizers of tantivy
.
- simple
- en_stem
- ja
Auto Trait Implementations
impl RefUnwindSafe for TokenizerManager
impl Send for TokenizerManager
impl Sync for TokenizerManager
impl Unpin for TokenizerManager
impl UnwindSafe for TokenizerManager
Blanket Implementations
impl<T> Any for T where
T: 'static + ?Sized,
[src]
T: 'static + ?Sized,
impl<T> Borrow<T> for T where
T: ?Sized,
[src]
T: ?Sized,
impl<T> BorrowMut<T> for T where
T: ?Sized,
[src]
T: ?Sized,
fn borrow_mut(&mut self) -> &mut T
[src]
impl<T> Downcast for T where
T: Any,
[src]
T: Any,
fn into_any(self: Box<T>) -> Box<dyn Any + 'static>
[src]
fn into_any_rc(self: Rc<T>) -> Rc<dyn Any + 'static>
[src]
fn as_any(&self) -> &(dyn Any + 'static)
[src]
fn as_any_mut(&mut self) -> &mut (dyn Any + 'static)
[src]
impl<T> DowncastSync for T where
T: Send + Sync + Any,
[src]
T: Send + Sync + Any,
impl<T> Erased for T
[src]
impl<T> From<T> for T
[src]
impl<T> Fruit for T where
T: Send + Downcast,
[src]
T: Send + Downcast,
impl<T, U> Into<U> for T where
U: From<T>,
[src]
U: From<T>,
impl<T> ToOwned for T where
T: Clone,
[src]
T: Clone,
type Owned = T
The resulting type after obtaining ownership.
fn to_owned(&self) -> T
[src]
fn clone_into(&self, target: &mut T)
[src]
impl<T, U> TryFrom<U> for T where
U: Into<T>,
[src]
U: Into<T>,
type Error = Infallible
The type returned in the event of a conversion error.
fn try_from(value: U) -> Result<T, <T as TryFrom<U>>::Error>
[src]
impl<T, U> TryInto<U> for T where
U: TryFrom<T>,
[src]
U: TryFrom<T>,
type Error = <U as TryFrom<T>>::Error
The type returned in the event of a conversion error.
fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>
[src]
impl<V, T> VZip<V> for T where
V: MultiLane<T>,
V: MultiLane<T>,