Struct TokenizeOptions 

pub struct TokenizeOptions {
    pub mode: Mode,
    pub ignore_case: bool,
    pub ignore_ranges: Vec<[usize; 2]>,
    pub ignore_identifiers: bool,
    pub ignore_literals: bool,
    pub ignore_annotations: bool,
    pub code_ignore_regexes: Vec<Regex>,
    pub strip_types_formats: HashSet<String>,
}

Options for the detection-path tokenizer.

Carries mode, case-folding flag, pre-parsed ignore-region byte ranges, and pre-compiled code-level regex patterns that skip matching tokens during detection.

Code-level ignore patterns (v4 ignorePattern) work by matching regex patterns against source text, collecting byte ranges of matches, and then filtering any token whose byte range overlaps a match — identical in effect to v4’s setupIgnorePatterns which injected Prism grammar tokens.

Fields

mode: Mode

ignore_case: bool

When true, token values are lowercased before hashing.

ignore_ranges: Vec<[usize; 2]>

Ignored byte ranges from jscpd:ignore-start / jscpd:ignore-end and code-level regex matches from ignorePattern. Each entry is [start_byte, end_byte).

ignore_identifiers: bool

Hash every identifier as $id so clones that differ only in variable, function or type names match (issue #998). Keywords are kept: the JavaScript tokenizer classifies them, other languages fall back to is_common_keyword.

ignore_literals: bool

Hash string literals as $str and numeric literals as $num.

ignore_annotations: bool

Drop @Name, @a.b.Name and @Name(...) annotation/decorator sequences before hashing, in formats listed by strips_annotations.

code_ignore_regexes: Vec<Regex>

Pre-compiled code-level regex patterns inherited from v4 ignorePattern. Before tokenization, these are matched against the source text and overlapping byte ranges are added to ignore_ranges.

strip_types_formats: HashSet<String>

Formats whose TypeScript-only syntax is stripped from the detection token stream (--cross-formats groups mixing TS with JS). Only typescript and tsx are meaningful here; empty by default so the standard detection path is untouched.

Implementations

impl TokenizeOptions

pub fn new(mode: Mode) -> Self

pub fn normalizes(&self) -> bool

True when any Type-2 normalization option is on.

pub fn with_code_ignore_patterns(mode: Mode, patterns: &[String]) -> Self

Build TokenizeOptions with pre-compiled regex patterns from string patterns. Invalid regex patterns are silently skipped.

Trait Implementations

impl Clone for TokenizeOptions

fn clone(&self) -> TokenizeOptions

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenizeOptions

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.