Module lexer 

Turns source text into tokens.

Besides the usual work, the lexer decides which line breaks end a statement (see filter_newlines), so the parser only ever sees significant ones.

Functions

filter_newlines

Drop the line breaks that don’t end a statement.

lex

Lex src, whose first byte sits at offset base of the enclosing file (non-zero when lexing an interpolated expression inside a string).