Module tokenizer 

Structs

TokenMap

A sub-format detection map produced by multi-format tokenizers.

TokenizeOptions

Options for the detection-path tokenizer.

Enums

Mode

Functions

push_token

Push a token into the detection output if it passes all filters.

tokenize

Tokenize source code in the given format with the given mode. Returns a Vec. Never panics on empty input — returns empty Vec.

tokenize_to_detection

Tokenize source code for the detection hot path.

tokenize_to_detection_maps

Tokenize source code into one or more format-specific detection maps.