pub fn lex(input: &str, profile: Profile) -> Vec<Token<'_>>Expand description
Lexes input into a flat token stream under profile.
Total: concatenating token.text for every returned token reproduces
input byte for byte. Input the family lexer does not recognise becomes
SyntaxKind::Error tokens rather than aborting the lex.
The profile decides only which registry words are active keywords: under
Profile::Typl the stream is identical to the E1 lexer’s; under
Profile::Ridl the nine ridl words lex to their keyword variants; under
Profile::Rsdl the eight rsdl words do.
Everything else — durations, @, the expr operators — is a family token
in both profiles; the parser draws the profile boundaries.