Struct tantivy::tokenizer::TextAnalyzer [−][src]
TextAnalyzer
tokenizes an input text into tokens and modifies the resulting TokenStream
.
It simply wraps a Tokenizer
and a list of TokenFilter
that are applied sequentially.
Implementations
impl TextAnalyzer
[src]
pub fn new<T: Tokenizer>(
tokenizer: T,
token_filters: Vec<BoxTokenFilter>
) -> TextAnalyzer
[src]
tokenizer: T,
token_filters: Vec<BoxTokenFilter>
) -> TextAnalyzer
Creates a new TextAnalyzer
given a tokenizer and a vector of BoxTokenFilter
.
When creating a TextAnalyzer
from a Tokenizer
alone, prefer using
TextAnalyzer::from(tokenizer)
.
pub fn filter<F: Into<BoxTokenFilter>>(mut self: Self, token_filter: F) -> Self
[src]
Appends a token filter to the current tokenizer.
The method consumes the current TokenStream
and returns a
new one.
Example
use tantivy::tokenizer::*; let en_stem = TextAnalyzer::from(SimpleTokenizer) .filter(RemoveLongFilter::limit(40)) .filter(LowerCaser) .filter(Stemmer::default());
pub fn token_stream_texts<'a>(&self, texts: &'a [&'a str]) -> BoxTokenStream<'a>
[src]
Tokenize an array&str
The resulting BoxTokenStream
is equivalent to what would be obtained if the &str were
one concatenated &str
, with an artificial position gap of 2
between the different fields
to prevent accidental PhraseQuery
to match accross two terms.
pub fn token_stream<'a>(&self, text: &'a str) -> BoxTokenStream<'a>
[src]
Creates a token stream for a given str
.
Trait Implementations
impl Clone for TextAnalyzer
[src]
fn clone(&self) -> Self
[src]
pub fn clone_from(&mut self, source: &Self)
1.0.0[src]
impl<T: Tokenizer> From<T> for TextAnalyzer
[src]
Auto Trait Implementations
impl !RefUnwindSafe for TextAnalyzer
[src]
impl Send for TextAnalyzer
[src]
impl Sync for TextAnalyzer
[src]
impl Unpin for TextAnalyzer
[src]
impl !UnwindSafe for TextAnalyzer
[src]
Blanket Implementations
impl<T> Any for T where
T: 'static + ?Sized,
[src]
T: 'static + ?Sized,
impl<T> Borrow<T> for T where
T: ?Sized,
[src]
T: ?Sized,
impl<T> BorrowMut<T> for T where
T: ?Sized,
[src]
T: ?Sized,
pub fn borrow_mut(&mut self) -> &mut T
[src]
impl<T> Downcast for T where
T: Any,
[src]
T: Any,
pub fn into_any(self: Box<T, Global>) -> Box<dyn Any + 'static, Global>
[src]
pub fn into_any_rc(self: Rc<T>) -> Rc<dyn Any + 'static>
[src]
pub fn as_any(&self) -> &(dyn Any + 'static)
[src]
pub fn as_any_mut(&mut self) -> &mut (dyn Any + 'static)
[src]
impl<T> DowncastSync for T where
T: Send + Sync + Any,
[src]
T: Send + Sync + Any,
impl<T> From<T> for T
[src]
impl<T> Fruit for T where
T: Send + Downcast,
[src]
T: Send + Downcast,
impl<T, U> Into<U> for T where
U: From<T>,
[src]
U: From<T>,
impl<T> Pointable for T
pub const ALIGN: usize
type Init = T
The type for initializers.
pub unsafe fn init(init: <T as Pointable>::Init) -> usize
pub unsafe fn deref<'a>(ptr: usize) -> &'a T
pub unsafe fn deref_mut<'a>(ptr: usize) -> &'a mut T
pub unsafe fn drop(ptr: usize)
impl<T> ToOwned for T where
T: Clone,
[src]
T: Clone,
type Owned = T
The resulting type after obtaining ownership.
pub fn to_owned(&self) -> T
[src]
pub fn clone_into(&self, target: &mut T)
[src]
impl<T, U> TryFrom<U> for T where
U: Into<T>,
[src]
U: Into<T>,
type Error = Infallible
The type returned in the event of a conversion error.
pub fn try_from(value: U) -> Result<T, <T as TryFrom<U>>::Error>
[src]
impl<T, U> TryInto<U> for T where
U: TryFrom<T>,
[src]
U: TryFrom<T>,
type Error = <U as TryFrom<T>>::Error
The type returned in the event of a conversion error.
pub fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>
[src]
impl<V, T> VZip<V> for T where
V: MultiLane<T>,
V: MultiLane<T>,