Function push_token 

pub fn push_token(
    tokens: &mut Vec<DetectionToken>,
    kind: TokenKind,
    value: &str,
    byte_start: usize,
    byte_end: usize,
    start: Location,
    end: Location,
    options: &TokenizeOptions,
)

Push a token into the detection output if it passes all filters.

Filtering happens here — at tokenize time — so the resulting Vec<DetectionToken> passed to detection is already minimal. Token values are not stored; only the pre-computed hash is kept.

The argument count is intentional: this function is a hot-path helper called from every tokenizer branch; grouping parameters into a struct would add an extra dereference per call.