Crate xgrammar 

Pure-Rust core of XGrammar — an open-source library for efficient, flexible, and portable structured generation.

XGrammar uses constrained decoding (grammar-guided generation) to enforce structural correctness of model output. It supports general context-free grammars, including JSON, regular expressions, and custom EBNF. A GrammarMatcher drives a non-deterministic pushdown automaton (NPDA) over a compiled grammar and tokenizer, producing token bitmasks that downstream inference engines apply to logits.

This crate is a from-scratch Rust reimplementation of the upstream xgrammar core. It carries no C/C++ dependency and builds with cargo alone on every supported target (macOS, Linux, Windows, iOS, wasm32).

Re-exports

pub use compiler::CompileError;

pub use compiler::CompiledGrammar;

pub use compiler::GrammarCompiler;

pub use config::SERIALIZATION_VERSION;

pub use config::get_serialization_version;

pub use converter::JsonFormat;

pub use converter::RegexError;

pub use converter::SchemaError;

pub use converter::SchemaErrorKind;

pub use converter::StructuralTagError;

pub use converter::StructuralTagItem;

pub use converter::generate_float_range_regex;

pub use converter::generate_float_range_regex_with_options;

pub use converter::generate_range_regex;

pub use converter::json_schema_to_ebnf;

pub use converter::regex_to_ebnf;

pub use grammar::DeserializeError;

pub use grammar::Grammar;

pub use matcher::BatchGrammarMatcher;

pub use matcher::BitmaskDlType;

pub use matcher::GrammarMatcher;

pub use matcher::MatcherError;

pub use matcher::allocate_token_bitmask;

pub use matcher::apply_token_bitmask_inplace_cpu;

pub use matcher::apply_token_bitmask_inplace_cpu_batch;

pub use matcher::get_bitmask_dl_type;

pub use matcher::get_bitmask_size;

pub use matcher::get_masked_tokens_from_bitmask;

pub use matcher::is_single_token_bitmask;

pub use matcher::reset_token_bitmask;

pub use support::RecursionError;

pub use support::get_max_recursion_depth;

pub use support::reset_recursion_depth;

pub use support::set_max_recursion_depth;

pub use testing::print_token_by_ids;

pub use tokenizer::HfMetadata;

pub use tokenizer::TokenizerError;

pub use tokenizer::TokenizerInfo;

pub use tokenizer::UnknownVocabType;

pub use tokenizer::VocabType;

pub use tokenizer::decode_token;

pub use tokenizer::decode_token_bytes;

pub use tokenizer::detect_metadata_from_hf;

Modules

compiler

Grammar compilation: preprocessing a grammar and tokenizer into a CompiledGrammar for use by GrammarMatcher, with a result cache.

config

Global configuration for XGrammar.

converter

Converters that turn external schema languages (regex, JSON Schema, structural tags, Lark) into XGrammar BNF grammars.

fsm

The finite-state-machine engine: edges, NFAs/DFAs, and the regex/trie/suffix-automaton builders that back grammar compilation.

functor

Grammar transformation passes — the “functors”: normalization, structure normalization, inlining, dead-code elimination, optimization, etc. Ported from

grammar

The grammar data model: the BNF abstract syntax tree (AST), its builder, and EBNF printing.

matcher

Grammar-guided generation: stateful matchers that drive constrained decoding.

parser

EBNF lexing and parsing into the BNF AST.

support

Foundational utilities: UTF-8 encoding, compact CSR arrays, dynamic bitsets, integer-set operations, hashing helpers, and the recursion-depth guard.

testing

Test and debug helpers re-exported for bindings and integration tests.

tokenizer

Tokenizer metadata for grammar-guided generation.

Constants

VERSION

The crate version, as declared in Cargo.toml.