Module grammar 

General structured-output grammar: a schema → flat-table → byte-FSM pipeline for GPU-resident constrained decoding (Workstream G).

G1 (this module) is CPU-only and the REFERENCE for everything downstream: the same compiled tables the CPU FSM here interprets are the buffers the WGSL interpreter (G2) will upload and read, and the serving layer (G3) drives this FSM at the per-column pick-override seam. Keeping one table-interpreter — rather than compiling a shader per schema — is the design’s core bet, and it is de-risked HERE: any JSON-Schema construct that cannot be encoded in the tables surfaces as a pure-Rust compile error before a single byte is generated.

Three pieces:

• json_schema — JsonSchemaTables::compile turns a serde_json::Value schema into flat u32/u8 tables (or refuses, naming the construct). See its docs for the exact array layout.
• json_fsm — JsonFsm walks those tables byte-by-byte over a fixed json_fsm::STATE_WORDS -word FsmState: step_byte, allowed_bytes (the constrained-decode mask), is_complete, and step_token.
• token_bytes — TokenByteTable maps a tokenizer’s token ids to raw output bytes (byte-level BPE via the engine’s vocab::build_vocab_blob, with an SPM/Unigram fallback), so the FSM can be advanced one whole token at a time.

Supported subset (OpenAI strict mode)

object/array/string/number/integer/boolean/null; properties + required with every property required and matched in the required array’s order (that array is the canonical key order — a parsed properties object cannot preserve declaration order without serde_json’s preserve_order, which this workspace does not enable); mandatory additionalProperties:false; enum/const over scalars; bounded arrays and strings; nesting depth ≤ 10; canonical output with no whitespace. pattern/anyOf/oneOf/allOf/$ref/format/minimum/… are refused loudly, with reserved node-kind discriminants (NodeKind::RESERVED_PATTERN et al.) held for the future regex-DFA extension.

Re-exports

pub use gpu::GpuGrammar;

pub use gpu::GrammarKernels;

pub use gpu::GrammarVocab;

pub use harness::GpuSchema;

pub use harness::GpuVocab;

pub use harness::SchemaMeta;

pub use json_fsm::FsmState;

pub use json_fsm::JsonFsm;

pub use json_schema::JsonSchemaTables;

pub use json_schema::NodeKind;

pub use token_bytes::TokenByteTable;

Modules

gpu

GPU-resident JSON-schema grammar: the WGSL transcription of [flat_step_byte] + a wgpu harness.

harness

Flat, GPU-uploadable encoding of the G1 tables + the contract the G2 WGSL interpreter fills in.

json_fsm

The fixed table-interpreter: a byte-driven pushdown automaton over JsonSchemaTables.

json_schema

JSON-Schema → flat table compiler (the “program” the byte-FSM interprets).

token_bytes

token id → raw bytes table, so the FSM can be walked one TOKEN at a time (the constrained decoder masks logits over token ids, then advances the grammar by the chosen token’s bytes).