Function tokenize_file 

pub fn tokenize_file(
    path: &Path,
    source: &str,
    skip_imports: bool,
) -> FileTokens

Tokenize a source file into a sequence of normalized tokens.

For Vue/Svelte SFC files, extracts <script> blocks first and tokenizes their content, mirroring the main analysis pipeline’s SFC handling. For Astro files, extracts frontmatter. For MDX files, extracts import/export statements.

When strip_types is true, TypeScript type annotations, interfaces, and type aliases are stripped from the token stream. This enables cross-language clone detection between .ts and .js files.

When skip_imports is true, ES import declarations are excluded from the token stream to reduce noise from sorted import blocks.