Module tokenizer 

Public tokenizer for GCTF assertion expressions.

Pipeline: text → tokenize_assertion() → Vec<Token> where each Token has a Span { start, end } with exact byte positions in the source.

This is used by:

• LSP semantic tokens — highlight operators, keywords, plugins, regexes
• Optimizer — safe string-literal-aware rule matching (4.3)
• Semantics — type-checking operators against TypeInfo

Structs

Span

Byte range in source text.

Token

A token with its source position.

Enums

TokenKind

Token kinds produced by the assertion tokenizer.

Functions

collect_identifiers

Collect all identifier tokens.

collect_operators

Collect all operator tokens.

collect_plugin_calls

Collect plugin call names with spans.

tokenize_assertion

Tokenize an assertion expression string into a list of tokens with exact byte positions.