Module token 

Lossless tokenizer.

Every byte of the input belongs to exactly one token, including whitespace and comments. Concatenating the text of all tokens reproduces the source exactly - a property the tests assert - so the same token stream serves a compiler front end, a formatter and an evidence extractor. Skipping trivia would make the stream cheaper and permanently unable to round-trip.

Structs

Token

One lexical unit and its exact position in the source.

Tokenizer

Streaming tokenizer over one source file.

Enums

Mode

How much of the source the stream carries.

TokenKind

What a token is, at the lexical level only.

Functions

tokenize

Tokenizes a whole source file losslessly.

tokenize_lite

Tokenizes a source file, dropping trivia.