Enum Token

pub enum Token {
    ApostrophenatedWord(String),
    CommaFormattedInteger(String),
    CommaFormattedRealNumber(String),
    Hashtag(String),
    HyphenatedWord(String),
    Integer(String),
    Punctuation(Punctuation),
    RealNumber(String),
    Url(String),
    UsernameMention(String),
    Word(String),
    Unknown(String),
}

A parsed token

Variants

ApostrophenatedWord(String)

An word with apostrophes matches “^‘?([A-Za-z]+’?)+$”. This should work for contractions, possessives, and plurals.

CommaFormattedInteger(String)

A comma-formatted integer. Like Integer, but has comma separators.

CommaFormattedRealNumber(String)

A comma-formatted real number. Like RealNumber, but has comma separators.

Hashtag(String)

Twitter-style hashtag, which matches ‘^#\w+$’.

HyphenatedWord(String)

A hyphenated word matches ‘^([A-Za-z]+-)+[A-Za-z]+$’. If the word isn’t in your dictionary, perhaps splitting by hyphens will help.

Integer(String)

A simple integer. Matches ‘\d+’

Punctuation(Punctuation)

A punctuation mark.

RealNumber(String)

A simple real number. Matches ‘\d+.\d+’

Url(String)

An http:// or https:// prefixed URL

UsernameMention(String)

Twitter-style username mention, which matches ‘^@\w+’

Word(String)

A word matches ‘^\w+$’.

Unknown(String)

A sequence that doesn’t match any other pattern. Catch-all.

Trait Implementations

impl Clone for Token

fn clone(&self) -> Token

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for Token

fn eq(&self, other: &Token) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for Token