Module rustpython_parser::lexer

This module takes care of lexing Python source text.

This means source code is scanned and translated into separate tokens. The rules governing what is and is not a valid token are defined in the Python reference guide section on Lexical analysis.

The primary function in this module is lex, which takes a string slice and returns an iterator over the tokens in the source code. The tokens are currently returned as a Result<Spanned, LexicalError>, where Spanned is a tuple containing the start and end TextSize and a Tok denoting the token.

Example

use rustpython_parser::{lexer::lex, Tok, Mode, StringKind};

let source = "x = 'RustPython'";
let tokens = lex(source, Mode::Module)
    .map(|tok| tok.expect("Failed to lex"))
    .collect::<Vec<_>>();

for (token, range) in tokens {
    println!(
        "{token:?}@{range:?}",
    );
}

Structs

• Lexer
  A lexer for Python source code.
• LexicalError
  Represents an error that occur during lexing and are returned by the parse_* functions in the iterator in the lexer implementation.

Enums

• LexicalErrorType
  Represents the different types of errors that can occur during lexing.

Statics

• KEYWORDS
  A map of keywords to their tokens.

Functions

• lex
  Create a new lexer from a source string.
• lex_starts_at
  Create a new lexer from a source string, starting at a given location. You probably want to use lex instead.

Type Aliases

• LexResult
  The result of lexing a token.
• Spanned
  Contains a Token along with its range.