Module token 

Token types.

A token is created during lexing, and records information about a matched Rule (or failure to match).

It is idiomatic to use ilex::token; and refer to types in this module with a token prefix. E.g., token::Any, token::Ident, and token::Stream.

The token types themselves are thin immutable references into a token::Stream, and should be passed around by value. They all implement Token.

Structs

Bracket

A bracket pair, such as matching ( ... ), which contains a substream of tokens.

Cursor

A cursor over a piece of a Stream.

Digital

A digital literal.

Eof

A special token for the end of a file.

Ident

A identifier, i.e., a self-delimiting word like foo or 黒猫.

Keyword

A keyword, i.e., an exact well-known string, such as +, class, and #define.

Quoted

A quoted literal.

Stream

A tree-like stream of tokens.

Switch

A token switch.

Enums

Any

A type-erased Token.

Content

A piece of a quoted literal.

Sign

A sign for a Digital literal.

Traits

FromRadix

A base 2 integer type of portable size that can be parsed from any radix.

Token

A token type. All types in ilex::token implement this trait.

Functions

switch

Creates a new, empty token switch.