Module trie 

Token trie for fast multi-pattern matching.

A 256-way trie that unifies all token types (comments, strings, keywords) into a single lookup structure. Combined with a process mask (bloom filter on first bytes), this skips ~90% of bytes in the hot loop.

Structs

TokenMatch

Result of a successful trie match.

TokenTrie

Fast token lookup trie. All token types for a language are inserted into one trie.

Enums

TokenType

Type of token matched by the trie.

Functions

build_from_language

Build a TokenTrie from a Language definition.

should_process

Fast check: could this byte possibly start a token?