Enum TokenKind 

pub enum TokenKind {
    Name,
    Int,
    Float,
    Complex,
    String,
    FStringStart,
    FStringMiddle,
    FStringEnd,
    TStringStart,
    TStringMiddle,
    TStringEnd,
    IpyEscapeCommand,
    Comment,
    Newline,
    NonLogicalNewline,
    Indent,
    Dedent,
    EndOfFile,
    Question,
    Exclamation,
    Lpar,
    Rpar,
    Lsqb,
    Rsqb,
    Colon,
    Comma,
    Semi,
    Plus,
    Minus,
    Star,
    Slash,
    Vbar,
    Amper,
    Less,
    Greater,
    Equal,
    Dot,
    Percent,
    Lbrace,
    Rbrace,
    EqEqual,
    NotEqual,
    LessEqual,
    GreaterEqual,
    Tilde,
    CircumFlex,
    LeftShift,
    RightShift,
    DoubleStar,
    DoubleStarEqual,
    PlusEqual,
    MinusEqual,
    StarEqual,
    SlashEqual,
    PercentEqual,
    AmperEqual,
    VbarEqual,
    CircumflexEqual,
    LeftShiftEqual,
    RightShiftEqual,
    DoubleSlash,
    DoubleSlashEqual,
    ColonEqual,
    At,
    AtEqual,
    Rarrow,
    Ellipsis,
    And,
    As,
    Assert,
    Async,
    Await,
    Break,
    Class,
    Continue,
    Def,
    Del,
    Elif,
    Else,
    Except,
    False,
    Finally,
    For,
    From,
    Global,
    If,
    Import,
    In,
    Is,
    Lambda,
    None,
    Nonlocal,
    Not,
    Or,
    Pass,
    Raise,
    Return,
    True,
    Try,
    While,
    With,
    Yield,
    Case,
    Lazy,
    Match,
    Type,
    Unknown,
}

A kind of a token.

Variants

Name

Token kind for a name, commonly known as an identifier.

Int

Token kind for an integer.

Float

Token kind for a floating point number.

Complex

Token kind for a complex number.

String

Token kind for a string.

FStringStart

Token kind for the start of an f-string. This includes the f/F/fr prefix and the opening quote(s).

FStringMiddle

Token kind that includes the portion of text inside the f-string that’s not part of the expression part and isn’t an opening or closing brace.

FStringEnd

Token kind for the end of an f-string. This includes the closing quote.

TStringStart

Token kind for the start of a t-string. This includes the t/T/tr prefix and the opening quote(s).

TStringMiddle

Token kind that includes the portion of text inside the t-string that’s not part of the interpolation part and isn’t an opening or closing brace.

TStringEnd

Token kind for the end of a t-string. This includes the closing quote.

IpyEscapeCommand

Token kind for a IPython escape command.

Comment

Token kind for a comment. These are filtered out of the token stream prior to parsing.

Newline

Token kind for a newline.

NonLogicalNewline

Token kind for a newline that is not a logical line break. These are filtered out of the token stream prior to parsing.

Indent

Token kind for an indent.

Dedent

Token kind for a dedent.

EndOfFile

Question

Token kind for a question mark ?.

Exclamation

Token kind for an exclamation mark !.

Lpar

Token kind for a left parenthesis (.

Rpar

Token kind for a right parenthesis ).

Lsqb

Token kind for a left square bracket [.

Rsqb

Token kind for a right square bracket ].

Colon

Token kind for a colon :.

Comma

Token kind for a comma ,.

Semi

Token kind for a semicolon ;.

Plus

Token kind for plus +.

Minus

Token kind for minus -.

Star

Token kind for star *.

Slash

Token kind for slash /.

Vbar

Token kind for vertical bar |.

Amper

Token kind for ampersand &.

Less

Token kind for less than <.

Greater

Token kind for greater than >.

Equal

Token kind for equal =.

Dot

Token kind for dot ..

Percent

Token kind for percent %.

Lbrace

Token kind for left bracket {.

Rbrace

Token kind for right bracket }.

EqEqual

Token kind for double equal ==.

NotEqual

Token kind for not equal !=.

LessEqual

Token kind for less than or equal <=.

GreaterEqual

Token kind for greater than or equal >=.

Tilde

Token kind for tilde ~.

CircumFlex

Token kind for caret ^.

LeftShift

Token kind for left shift <<.

RightShift

Token kind for right shift >>.

DoubleStar

Token kind for double star **.

DoubleStarEqual

Token kind for double star equal **=.

PlusEqual

Token kind for plus equal +=.

MinusEqual

Token kind for minus equal -=.

StarEqual

Token kind for star equal *=.

SlashEqual

Token kind for slash equal /=.

PercentEqual

Token kind for percent equal %=.

AmperEqual

Token kind for ampersand equal &=.

VbarEqual

Token kind for vertical bar equal |=.

CircumflexEqual

Token kind for caret equal ^=.

LeftShiftEqual

Token kind for left shift equal <<=.

RightShiftEqual

Token kind for right shift equal >>=.

DoubleSlash

Token kind for double slash //.

DoubleSlashEqual

Token kind for double slash equal //=.

ColonEqual

Token kind for colon equal :=.

At

Token kind for at @.

AtEqual

Token kind for at equal @=.

Rarrow

Token kind for arrow ->.

Ellipsis

Token kind for ellipsis ....

And

As

Assert

Async

Await

Break

Class

Continue

Def

Del

Elif

Else

Except

False

Finally

For

From

Global

If

Import

In

Is

Lambda

None

Nonlocal

Not

Or

Pass

Raise

Return

True

Try

While

With

Yield

Case

Lazy

Match

Type

Unknown

Implementations

impl TokenKind

pub const fn is_eof(self) -> bool

Returns true if this is an end of file token.

pub const fn is_any_newline(self) -> bool

Returns true if this is either a newline or non-logical newline token.

pub fn is_keyword(self) -> bool

Returns true if the token is a keyword (including soft keywords).

See also is_soft_keyword, is_non_soft_keyword.

pub fn is_soft_keyword(self) -> bool

Returns true if the token is strictly a soft keyword.

See also is_keyword, is_non_soft_keyword.

pub fn is_non_soft_keyword(self) -> bool

Returns true if the token is strictly a non-soft keyword.

See also is_keyword, is_soft_keyword.

pub const fn is_operator(self) -> bool

pub const fn is_singleton(self) -> bool

Returns true if this is a singleton token i.e., True, False, or None.

pub const fn is_trivia(&self) -> bool

Returns true if this is a trivia token i.e., a comment or a non-logical newline.

pub const fn is_comment(&self) -> bool

Returns true if this is a comment token.

pub const fn is_arithmetic(self) -> bool

pub const fn is_bitwise_or_shift(self) -> bool

pub const fn is_unary_arithmetic_operator(self) -> bool

Returns true if the current token is a unary arithmetic operator.

pub const fn is_interpolated_string_end(self) -> bool

pub const fn as_unary_arithmetic_operator(self) -> Option<UnaryOp>

Returns the UnaryOp that corresponds to this token kind, if it is a unary arithmetic operator, otherwise return None.

Use as_unary_operator to match against any unary operator.

pub const fn as_unary_operator(self) -> Option<UnaryOp>

Returns the UnaryOp that corresponds to this token kind, if it is a unary operator, otherwise return None.

Use as_unary_arithmetic_operator to match against only an arithmetic unary operator.

pub const fn as_bool_operator(self) -> Option<BoolOp>

Returns the BoolOp that corresponds to this token kind, if it is a boolean operator, otherwise return None.

pub const fn as_binary_operator(self) -> Option<Operator>

Returns the binary Operator that corresponds to the current token, if it’s a binary operator, otherwise return None.

Use as_augmented_assign_operator to match against an augmented assignment token.

pub const fn as_augmented_assign_operator(self) -> Option<Operator>

Returns the Operator that corresponds to this token kind, if it is an augmented assignment operator, or None otherwise.

Trait Implementations

impl Clone for TokenKind

fn clone(&self) -> TokenKind

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenKind

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl Display for TokenKind

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl From<BoolOp> for TokenKind

fn from(op: BoolOp) -> TokenKind

Converts to this type from the input type.

impl From<Operator> for TokenKind

fn from(op: Operator) -> TokenKind

Converts to this type from the input type.

impl From<UnaryOp> for TokenKind

fn from(op: UnaryOp) -> TokenKind

Converts to this type from the input type.

impl GetSize for TokenKind

fn get_heap_size(&self) -> usize

Determines how many bytes this object occupies inside the heap.

fn get_heap_size_with_tracker<TRACKER>( &self, tracker: TRACKER, ) -> (usize, TRACKER)

where TRACKER: GetSizeTracker,

Determines how many bytes this object occupies inside the heap while using a tracker.

fn get_stack_size() -> usize

Determines how may bytes this object occupies inside the stack.

fn get_size(&self) -> usize

Determines the total size of the object.

fn get_size_with_tracker<T>(&self, tracker: T) -> (usize, T)

where T: GetSizeTracker,

Determines the total size of the object while using a tracker.

impl Hash for TokenKind

fn hash<__H>(&self, state: &mut __H)

where __H: Hasher,

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H)

where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher.

impl Ord for TokenKind

fn cmp(&self, other: &TokenKind) -> Ordering

This method returns an Ordering between self and other.

fn max(self, other: Self) -> Self

where Self: Sized,

Compares and returns the maximum of two values.

fn min(self, other: Self) -> Self

where Self: Sized,

Compares and returns the minimum of two values.

fn clamp(self, min: Self, max: Self) -> Self

where Self: Sized,

Restrict a value to a certain interval.

impl PartialEq for TokenKind

fn eq(&self, other: &TokenKind) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl PartialOrd for TokenKind

fn partial_cmp(&self, other: &TokenKind) -> Option<Ordering>

This method returns an ordering between self and other values if one exists.

fn lt(&self, other: &Rhs) -> bool

Tests less than (for self and other) and is used by the < operator.

fn le(&self, other: &Rhs) -> bool

Tests less than or equal to (for self and other) and is used by the <= operator.

fn gt(&self, other: &Rhs) -> bool

Tests greater than (for self and other) and is used by the > operator.

fn ge(&self, other: &Rhs) -> bool

Tests greater than or equal to (for self and other) and is used by the >= operator.

impl Copy for TokenKind

impl Eq for TokenKind

impl StructuralPartialEq for TokenKind