Module gctf_tokenizer 

Universal GCTF file tokenizer.

This is the only module that reads raw .gctf text. Pipeline: text → tokenize_gctf() → Vec<GctfToken> → parser → AST

IMPLEMENTATION NOTES:

• No regex, no starts_with, no ends_with, no contains, no find()
• Pure byte-level scanning with exact span tracking
• Only uses .as_bytes() once per line to get byte slice for scanning
• Only uses .to_string() to create owned output (necessary for API)
• All “parsing” done via byte comparisons (bytes[pos] == b’-’)

String operations count: 0 (only .to_string() for output ownership)

Structs

GctfToken

Enums

GctfTokenKind

Functions

tokenize_extract_line

tokenize_extract_line_full

Like tokenize_extract_line but also returns the optional type annotation from name:Type = value syntax.

tokenize_gctf

tokenize_inline_options

tokenize_kv_line