Struct tantivy::tokenizer::Token [−][src]
pub struct Token { pub offset_from: usize, pub offset_to: usize, pub position: usize, pub text: String, pub position_length: usize, }
Expand description
Token
Fields
offset_from: usize
Offset (byte index) of the first character of the token. Offsets shall not be modified by token filters.
offset_to: usize
Offset (byte index) of the last character of the token + 1. The text that generated the token should be obtained by &text[token.offset_from..token.offset_to]
position: usize
Position, expressed in number of tokens.
text: String
Actual text content of the token.
position_length: usize
Is the length expressed in term of number of original tokens.
Trait Implementations
fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error> where
__D: Deserializer<'de>,
[src]
fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error> where
__D: Deserializer<'de>,
[src]Deserialize this value from the given Serde deserializer. Read more
Auto Trait Implementations
impl RefUnwindSafe for Token
impl UnwindSafe for Token
Blanket Implementations
Mutably borrows from an owned value. Read more
Convert Box<dyn Trait>
(where Trait: Downcast
) to Box<dyn Any>
. Box<dyn Any>
can
then be further downcast
into Box<ConcreteType>
where ConcreteType
implements Trait
. Read more
Convert Rc<Trait>
(where Trait: Downcast
) to Rc<Any>
. Rc<Any>
can then be
further downcast
into Rc<ConcreteType>
where ConcreteType
implements Trait
. Read more
Convert &Trait
(where Trait: Downcast
) to &Any
. This is needed since Rust cannot
generate &Any
’s vtable from &Trait
’s. Read more
Convert &mut Trait
(where Trait: Downcast
) to &Any
. This is needed since Rust cannot
generate &mut Any
’s vtable from &mut Trait
’s. Read more
pub fn vzip(self) -> V