Function tokenize_file 

pub fn tokenize_file(path: &Path, source: &str) -> FileTokens

Tokenize a source file into a sequence of normalized tokens.

For Vue/Svelte SFC files, extracts <script> blocks first and tokenizes their content, mirroring the main analysis pipeline’s SFC handling. For Astro files, extracts frontmatter. For MDX files, extracts import/export statements.

When strip_types is true, TypeScript type annotations, interfaces, and type aliases are stripped from the token stream. This enables cross-language clone detection between .ts and .js files.