Module ilex::token

Token types.

A token is created during lexing, and records information about a matched Rule (or failure to match).

It is idiomatic to use ilex::token; and refer to types in this module with a token prefix. E.g., token::Any, token::Ident, and token::Stream.

The token types themselves are thin immutable references into a token::Stream, and should be passed around by value. They all implement Token.

Structs

• Bracket
  A bracket pair, such as matching ( ... ), which contains a substream of tokens.
• Cursor
  A cursor over a piece of a Stream.
• Digital
  A digital literal.
• Eof
  A special token for the end of a file.
• Ident
  A identifier, i.e., a self-delimiting word like foo or 黒猫.
• Keyword
  A keyword, i.e., an exact well-known string, such as +, class, and #define.
• Quoted
  A quoted literal.
• Stream
  A tree-like stream of tokens.
• Switch
  A token switch.

Enums

• Any
  A type-erased Token.
• Content
  A piece of a quoted literal.
• Sign
  A sign for a Digital literal.

Traits

• FromRadix
  A base 2 integer type of portable size that can be parsed from any radix.
• Token
  A token type. All types in ilex::token implement this trait.

Functions

• switch
  Creates a new, empty token switch.