Module tokenizer 

Structs

TokenMap

A sub-format detection map produced by multi-format tokenizers.

TokenizeOptions

Options for the detection-path tokenizer.

Enums

Mode

Functions

code_ignore_ranges

Compute byte ranges of all regex matches against source text. Used to populate ignore_ranges from ignorePattern regexes before tokenization, matching v4 semantics where regex patterns match against source text regions (not individual token values).

push_token

Push a token into the detection output if it passes all filters.

tokenize

Tokenize source code in the given format with the given mode. Returns a Vec. Never panics on empty input — returns empty Vec.

tokenize_to_detection

Tokenize source code for the detection hot path.

tokenize_to_detection_maps

Tokenize source code into one or more format-specific detection maps.