Module tokens 

Token tape types.

The token tape is the lossless representation of the source. Every token carries a byte-offset span; trivia (whitespace, comments) is preserved verbatim in a side-table. Round-tripping is:

reconstruct(token_tape(input)) == input   // byte-for-byte

This contract is enforced by the proptest in tests/conformance.rs.

Structs

Token

A single syntactic token in the token tape.

TokenTape

An ordered sequence of tokens representing the full lexed source.

TriviaTable

Maps each token index to the trivia that precedes it.

Enums

TokenKind

Discriminator for a syntactic token.

Trivia

A piece of trivia — whitespace, comments, or other non-token source text that must be preserved for lossless round-tripping.