Module parser 

The GBNF parser, transcribed from llama_grammar_parser in llama.cpp’s src/llama-grammar.cpp.

It compiles grammar text into a flat rule table. Every construct that is not a literal character, a character class, a rule reference or a token is rewritten into extra synthesized rules at parse time, so the stack machine in super::machine only ever sees five element kinds.

The rewrites are upstream’s, quoted from the comment in parse_sequence:

S{m,n} --> S S S (m times) S'(n-m)
           S'(x)   ::= S S'(x-1) |
           S'(1)   ::= S |
S{m,}  --> S S S (m times) S'
           S'      ::= S S' |
S*     --> S{0,}   -->  S'  ::= S S' |
S+     --> S{1,}   -->  S S'    with S' ::= S S' |
S?     --> S{0,1}  -->  S'  ::= S |

Getting these exactly right matters more than it looks: the synthesized rule ids are observable, because S' is named <rule>_<id> and the id is the symbol count at the time it is generated. Two parsers that accept the same language can still build different rule tables, and llama.cpp’s own tests/test-grammar-parser.cpp pins the tables, not the language. Those pinned tables are transcribed in [super::parser_tests].

Structs

ParsedGrammar

A parsed grammar: the rule table plus the symbol names that produced it.

Constants

MAX_REPETITION_THRESHOLD

MAX_REPETITION_THRESHOLD: the ceiling on both a single repetition count and on the running product of nested repetitions.

Traits

GrammarVocab

Resolves the <name> form of a grammar token element to a token id.

Functions

parse

Parse GBNF text with no vocabulary. The <name> token form is refused by name; <[id]> works.

parse_with_vocab

Parse GBNF text, resolving <name> token elements through vocab.