pub fn tokenize(
source: &str,
) -> Result<Vec<Spanned<Token>>, Vec<Spanned<LexerError>>>Expand description
Tokenize kaish source into spanned tokens.
Pipeline: one composed scan (heredocs + arithmetic extracted with full
quote/escape/comment awareness, complete replacement table) → logos →
positional marker resolution → span correction back to original
coordinates → fusion passes (flag-metachar, colon, glob) with
verbatim-slice text. All spans — including HereDoc tokens and
everything after them — are exact original-source byte ranges.