[−][src]Struct notmecab::LexerToken
Fields
cost: i64
Used internally during lattice pathfinding.
start: usize
Location, in codepoints, of the surface of this LexerToken in the string it was parsed from.
end: usize
Corresponding ending location, in codepoints. Exclusive. (i.e. when start+1 == end, the LexerToken's surface is one codepoint long)
kind: TokenType
Origin of token. BOS and UNK are virtual origins ("beginning/ending-of-string" and "unknown", respectively). Normal means it came from the mecab dictionary.
The BOS (beginning/ending-of-string) tokens are stripped away in parse_to_lexertokens.
original_id: u32
Unique identifier of what specific lexeme realization this is, from the mecab dictionary. changes between dictionary versions.
Trait Implementations
impl Clone for LexerToken
[src]
fn clone(&self) -> LexerToken
[src]
fn clone_from(&mut self, source: &Self)
1.0.0[src]
Performs copy-assignment from source
. Read more
impl Debug for LexerToken
[src]
Auto Trait Implementations
impl Send for LexerToken
impl Sync for LexerToken
Blanket Implementations
impl<T, U> Into for T where
U: From<T>,
[src]
U: From<T>,
impl<T> ToOwned for T where
T: Clone,
[src]
T: Clone,
impl<T> From for T
[src]
impl<T, U> TryFrom for T where
U: Into<T>,
[src]
U: Into<T>,
type Error = Infallible
The type returned in the event of a conversion error.
fn try_from(value: U) -> Result<T, <T as TryFrom<U>>::Error>
[src]
impl<T> Borrow for T where
T: ?Sized,
[src]
T: ?Sized,
impl<T> Any for T where
T: 'static + ?Sized,
[src]
T: 'static + ?Sized,
impl<T> BorrowMut for T where
T: ?Sized,
[src]
T: ?Sized,
fn borrow_mut(&mut self) -> &mut T
[src]
impl<T, U> TryInto for T where
U: TryFrom<T>,
[src]
U: TryFrom<T>,