Module token 

Structs

TokenId

TokenSink

Mutable append-only view used by a token source.

TokenSourceError

A diagnostic buffered by a token source while it was producing tokens.

TokenSpec

The fields emitted for one token.

TokenStore

Canonical compact storage for every token associated with one token stream.

TokenStoreError

TokenView

Borrowing public view of one canonical token-store record.

Enums

TokenChannel

Constants

DEFAULT_CHANNEL

HIDDEN_CHANNEL

INVALID_TOKEN_TYPE

MAX_TOKEN_OFFSET

Largest source or location offset accepted by the compact token store.

TOKEN_EOF

Traits

Token

TokenSource