Skip to main content

Module lex

vyre_libs::parsing::python

Module lex 

Source
Expand description

Python byte lexer.

Constants§

TOK_AS
as keyword token.
TOK_ASYNC
async keyword token.
TOK_AT
@ token.
TOK_AWAIT
await keyword token.
TOK_CASE
case keyword token.
TOK_CLASS
class keyword token.
TOK_COLON
: token.
TOK_COMMA
, token.
TOK_COMMENT
Comment token.
TOK_DEF
def keyword token.
TOK_DOT
. token.
TOK_EQ
= token.
TOK_EXCEPT
except keyword token.
TOK_FROM
from keyword token.
TOK_IDENTIFIER
Identifier token.
TOK_IMPORT
import keyword token.
TOK_LBRACE
{ token.
TOK_LBRACKET
[ token.
TOK_LPAREN
( token.
TOK_MATCH
match keyword token.
TOK_NEWLINE
Newline token.
TOK_NONE
Sparse-token sentinel: non-token byte positions stay zeroed.
TOK_NUMBER
Number literal token.
TOK_RBRACE
} token.
TOK_RBRACKET
] token.
TOK_RPAREN
) token.
TOK_STAR
* token.
TOK_STRING
String literal token.
TOK_WITH
with keyword token.

Functions§

python312_lexer
GPU Python 3.12 sparse lexer.