Skip to main content

tokenize_js_stripped

Function tokenize_js_stripped 

Source
pub fn tokenize_js_stripped(source: &str, format: &str) -> Vec<Token>
Expand description

Tokenize TS/TSX source with erasable TypeScript-only syntax stripped from the token stream (cross-format detection). Token locations still reference the original source. Never panics.

Sources that fail to parse fall back to the word-split tokenizer WITHOUT stripping — they simply won’t cross-match JavaScript files.