Skip to main content

parse_source_with_tokens

Function parse_source_with_tokens 

Source
pub fn parse_source_with_tokens<R: Send>(
    file: &RelPath,
    text: &str,
    visit: impl FnOnce(&File, TokenStream) -> R + Send,
) -> Result<R, LoadError>
Expand description

Parse once and give the visitor the syntax and its original native tokens.

The same file token and recursion checks as parse_source run before the worker starts. Both native values stay on that worker; only Send results return. The AST is parsed from one shared stream clone; native parsing can copy token payloads. The complete source is not lexed a second time, and caller-thread spans are never invalidated.

Byte ranges index the original text, including after a leading BOM or shebang. The BOM is handled by the native lexer. Language-ignored shebang bytes are replaced with same-length whitespace solely for lexical input; the original header is retained in syn::File::shebang. No file is edited and the header is never executed. Native parser/token/TLS heap and visitor allocations retain their separate resource contracts.

§Errors

Returns the located native budget, storage, syntax or worker-start refusal. The visitor is never called after a failed admission, lex or parse.

§Panics

Resumes parser or visitor panics on the caller with their original payload.