Enum Tokenizer 

pub enum Tokenizer {
    Blank,
    Camel,
    Class,
    Punct,
    Segment(SegmentLanguage),
}

Variants

Blank

Camel

Class

Punct

Segment(SegmentLanguage)

Splits a span into words using a morphological dictionary.

Unlike the character-class tokenizers, which decide a boundary from the pair of characters around it, this one needs the whole span at once: the boundaries come from a lattice search over dictionary entries. It therefore runs after the character-class tokenizers have produced spans, segmenting each one further.

Implementations

impl Tokenizer

pub fn walk_revisioned_variant_name( wire_revision: u16, discriminant: u32, ) -> Option<&'static str>

Resolve a wire discriminant at wire_revision to the variant identifier at that revision. Returns None if no such variant exists at that revision.

Generated by the revisioned derive.

pub fn walk_revisioned_variant_table( wire_revision: u16, ) -> &'static [(&'static str, u32)]

Variant name + discriminant pairs at wire_revision. Returns an empty slice if the revision is unknown. Generated by the revisioned derive.

impl Tokenizer

pub const REVISION: u16 = 2u16

The revision number of this type, as declared via #[revisioned(revision = N)].

Trait Implementations

impl Clone for Tokenizer

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for Tokenizer

impl Debug for Tokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl DeserializeRevisioned for Tokenizer

fn deserialize_revisioned<R: Read>(reader: &mut R) -> Result<Self, Error>

Deserializes a new instance of the struct from the specified reader.

impl Display for Tokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for Tokenizer

impl From<Tokenizer> for Tokenizer

fn from(v: Tokenizer) -> Self

Converts to this type from the input type.

impl From<Tokenizer> for Tokenizer

fn from(v: Tokenizer) -> Self

Converts to this type from the input type.

impl Hash for Tokenizer

fn hash<__H: Hasher>(&self, state: &mut __H)

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H)

where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher.

impl PartialEq for Tokenizer

fn eq(&self, other: &Self) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl Revisioned for Tokenizer

fn revision() -> u16

Returns the current revision of this type.

fn type_id() -> TypeId

where Self: 'static,

Returns the type id of this type.

impl SerializeRevisioned for Tokenizer

fn serialize_revisioned<W: Write>(&self, writer: &mut W) -> Result<(), Error>

Serializes the struct using the specified writer.

impl SkipCheckRevisioned for Tokenizer

fn skip_check_revisioned<R: Read>(reader: &mut R) -> Result<(), Error>

impl SkipRevisioned for Tokenizer

fn skip_revisioned<R: Read>(reader: &mut R) -> Result<(), Error>

fn skip_revisioned_slice(reader: &mut SliceReader<'_>) -> Result<(), Error>

Consume through bytes backed by SliceReader without allocating a buffer for bulk segments.

impl StructuralPartialEq for Tokenizer

impl WalkRevisioned for Tokenizer

type Walker<'r, R: BorrowedReader + 'r> = TokenizerWalker<'r, R>

Walker produced by walk_revisioned.

fn walk_revisioned<'r, R: BorrowedReader>( reader: &'r mut R, ) -> Result<Self::Walker<'r, R>, Error>

Read the type’s revision-level prefix and return a walker positioned at the start of the body.