pub fn tokenize_algol_spanned_with_budget(
source: &str,
budget: &mut DecodeBudget,
) -> Result<Vec<SpannedToken>>Expand description
Tokenizes Algol source into spanned tokens under an explicit budget.
Scans identifiers, numbers, string literals, operator runs, parentheses, and commas, attaching leading whitespace and comment trivia to each token. The budget bounds trivia and string sizes; an unterminated block comment or an unexpected character is reported as an error.