Expand description
Common utilities for working with tokens and lexers.
Re-exports§
pub use generic_arraydeque;pub use generic_arraydeque::typenum;
Modules§
- cmp
- A module for custom comparing traits.
- container
smallvec - A module for container types with small size optimizations.
- delimiter
- Common delimiters used in lexing and parsing.
- human_
display - A module for displaying in a human-friendly way.
- knowledge
- Common knowledge types for lexing and parsing.
- marker
- Marker types used in various utilities.
- recursion_
tracker - Trackers for preventing infinite recursion in parsers.
- sdl_
display - A module for displaying in SDL.
- syntax_
tree_ display - A module for displaying in syntax trees.
- token_
tracker - A token tracker for tracking tokens in a lexer.
- tracker
- A tracker for tracking recursion depth and tokens.
Structs§
- Delimited
- A value delimited by opening and closing markers with source location tracking.
- Escaped
Lexeme - A generic escape sequence representation using a lexeme.
- Generic
Array Deque - A fixed-capacity, stack-allocated double-ended queue (deque) backed by
GenericArray. - Located
- A value with complete location information: both which source and where in that source.
- Message
- Feature-aware message container with a unified API.
- Multi
Char Escape - A multi-character escape sequence representation.
- Positioned
Char - A character paired with its byte position in the source input.
- Simple
Span - A lightweight span representing a range of positions in source input.
- Single
Char Escape - A single-character escape sequence representation.
- Sliced
- A value paired with its slice metadata.
- Spanned
- A value paired with its source location span.
Enums§
- Expected
- An enumeration representing expected tokens or values in parsing contexts.
- Lexeme
- A compact, zero-copy description of a lexeme in source code.
Traits§
- AsSpan
- Enables accessing the source span of a parsed element.
- CharLen
- A trait for character-like types that can report their encoded length in bytes.
- Into
Components - Enables destructuring a parsed element into its constituent components.
- Into
Equivalent - A trait for converting into an equivalent type.
- Into
Span - Enables consuming a parsed element to extract its source span.
- IsAscii
Char - A trait for checking if a token is an ASCII character.
- ToEquivalent
- A trait for converting to an equivalent type.