Module structural_tokenizer 

Structural tokenizer treating idiomatic multi-token spans as single motifs.

Structs

StructuralToken

Enums

TokenKind

Functions

structural_tokenize

Tokenize source into weighted structural tokens (motifs, keywords, literals, …).