Module tokenize 

Structs

FileTokens

Result of tokenizing a source file.

SourceToken

A single token extracted from the AST with its source location.

Enums

KeywordType

JavaScript/TypeScript keyword types.

OperatorType

Operator categories.

PunctuationType

Punctuation / delimiter types.

TokenKind

Normalized token types for clone detection.

Functions

tokenize_file

Tokenize a source file into a sequence of normalized tokens.

tokenize_file_cross_language

Tokenize a source file with optional type stripping for cross-language detection.