Crate oq3_lexer

Low-level OpenQASM3 lexer.

This is largely copied from rustc_lexer, the lexer for the rust compiler. rustc_lexer is avaiable as a crate, and rust-analyzer uses this crate for lexing.

The main entity of this crate is the TokenKind enum which represents common lexeme types.

Modules

• unescape
  Utilities for validating string and char literals and turning them into values they represent.

Structs

• Cursor
  Peekable iterator over a char sequence.
• Token
  Parsed token. It doesn’t contain information about data that has been parsed, only the type of the token and its size.

Enums

• Base
  Base of numeric literal encoding according to its prefix.
• LiteralKind
  Enum representing the literal types supported by the lexer.
• TokenKind
  Enum representing common lexeme types.

Functions

• is_id_continue
  For OQ3 we take Rust’s rules on valid identifiers as a starting point. True if c is valid as a non-first character of an identifier. See Rust language reference for a formal definition of valid identifier name.
• is_id_start
  For OQ3 we take Rust’s rules on valid identifiers as a starting point. True if c is valid as a first character of an identifier. See Rust language reference for a formal definition of valid identifier name.
• is_ident
  The passed string is lexically an identifier.
• is_whitespace
  True if c is considered a whitespace according to Rust language definition. See Rust language reference for definitions of these classes.
• tokenize
  Creates an iterator that produces tokens from the input string.