Module lexer 

Tokenizer for natural-language time expressions.

The parsers used to run directly over &str, fusing lexing and parsing into a single character-level pass — a “larser”. That design makes keyword matching prefix-based: "day" matches inside "days", "m" inside "min", so every alternation has to be hand-ordered longest-first, a convention that fails silently when broken.

Splitting the lexer out removes that class of bug structurally. A word is consumed maximally and then compared as a whole, so a keyword can never match part of a longer word regardless of the order alternatives appear in.

Tokenising does not by itself fix the phrase-level version of the same hazard — choice still commits to the first alternative that succeeds, so a bare tomorrow could shadow tomorrow morning. That one is handled in the grammar rather than here, by left-factoring the shared prefix and making the remainder optional (day_reference().then(part_of_day().or_not())), and inside keyword tables by phrases_ci, which sorts its entries so the table’s source order stays irrelevant.

Enums

Token

A single lexical unit of a time expression.

Functions

lex

Split input into tokens, each paired with its byte span in the source.