Module tokenizer 

Tokenizer for ISO Prolog source.

Ported from patch-prolog’s tokenizer.rs, split into focused submodules:

• [token]: TokenKind / Token value types and their Display.
• [chars]: unquoted atoms and variables.
• [numbers]: integer / float literals.
• [quoted]: single-quoted atoms.
• [symbols]: multi-character symbolic operator dispatch.

The driver (Tokenizer::next_token) handles whitespace/comments, single-character punctuation, and dispatches everything else.

Structs

Token

Tokenizer

Enums

TokenKind

Token types for Edinburgh Prolog.