Expand description
JavaScript tokenizer.
Produces a flat token stream ending in Eof. Unlike Python, JS is not
indentation-sensitive: blocks are brace-delimited and statements are
semicolon-terminated, with Automatic Semicolon Insertion (ASI) filling in
for newline-terminated statements. Each token records whether a line break
preceded it (newline_before) so the parser can apply ASI. // and /* */
comments are stripped here. Template literals are emitted as a single
Template token carrying the cooked quasis plus the raw source of each
${...} field; the parser recursively parses those fields.
Structs§
- Token
- A token plus its 1-based source line and whether a newline preceded it.
Enums§
- Tok
- A lexical token.
Functions§
- lex
- Tokenize
srcinto a token stream ending inEof.