Crate tokenizer_py

source ·

Structs§

  • Tokenizer
    A struct that can tokenize a string into tokens.

Enums§

  • Token
    An enumeration of Python tokens.
  • TokenizerError
    An enumeration of possible errors that can occur during tokenization.

Constants§

  • OPERATORS
    The constant OPERATORS contains a string that lists all possible operators that can be used in expressions.