Module guided 

Regex-guided decoding — hard token masking via DFA.

Given a regex pattern and a tokenizer vocab, build a DFA and at each sampling step compute which tokens can extend the currently accepted prefix without leaving the language. Invalid tokens get -INFINITY so the downstream sampler cannot pick them, regardless of temperature or top-k/top-p.

This is the “outlines”-style approach: convert the constraint to a finite automaton, walk it byte-by-byte per token to decide validity. No schema → regex transformation here — that belongs a layer up (see ResponseFormat::JsonSchema handling).

Design notes

• The DFA is built once at request admission — regex compilation is the expensive step (~1-5 ms for short patterns, scales with ambiguity).
• Per-step cost is O(vocab_size · avg_token_bytes). For a 150k vocab with ~5 byte tokens that’s ~750k state transitions per sampling step. Fine for single requests; we’ll add a cached (state, token) → (valid, next_state) transition table if this becomes a bottleneck.
• End-of-string: once the DFA can accept, EOS becomes a valid choice. If the pattern is “open” (e.g. .*) EOS is always allowed.

Structs

RegexGuidedProcessor

Hard-mask regex constraint processor.