Skip to main content

Module token

ruff_python_ast

Module token 

Source
Expand description

Token kinds for Python source code created by the lexer and consumed by the ruff_python_parser.

This module defines the tokens that the lexer recognizes. The tokens are loosely based on the token definitions found in the CPython source.

Structs§

Token
TokenFlags
TokenIterWithContext
An iterator over the Tokens with context.
Tokens
Tokens represents a vector of lexed Token.

Enums§

TokenAt
A token that encloses a given offset or ends exactly at it.
TokenKind
A kind of a token.

Functions§

parentheses_iterator
Returns an iterator over the ranges of the optional parentheses surrounding an expression.
parenthesized_range
Returns the TextRange of a given expression including parentheses, if the expression is parenthesized; or None, if the expression is not parenthesized.