1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
//! # schoolmarm
//!
//! GBNF grammar-constrained decoding for LLM inference, ported from llama.cpp.
//!
//! This crate provides grammar-constrained sampling for autoregressive language model
//! inference. Given a GBNF grammar string and a vocabulary of token strings, it produces
//! bitmasks of allowed tokens at each generation step.
//!
//! ## Usage
//!
//! ```rust
//! use schoolmarm::{Grammar, GrammarState};
//!
//! // Parse a grammar
//! let grammar = Grammar::new(r#"root ::= "hello" | "world""#).unwrap();
//!
//! // Create runtime state
//! let mut state = GrammarState::new(grammar).unwrap();
//!
//! // Get allowed tokens for your vocabulary
//! let vocab = vec!["hello", "world", "foo", "hel"];
//! let allowed = state.allowed_tokens(&vocab);
//! // allowed = [true, true, false, true]
//!
//! // Accept a token and advance state
//! state.accept_token("hello").unwrap();
//! assert!(state.is_accepting());
//! ```
//!
//! ## GBNF Format
//!
//! GBNF (GGML BNF) is an extended BNF notation for defining formal grammars.
//! See the [llama.cpp GBNF documentation](https://github.com/ggml-org/llama.cpp/blob/master/grammars/README.md)
//! for the full specification.
//!
//! Supported features:
//! - Literals: `"hello"`
//! - Character ranges: `[a-z]`, `[^0-9]`, `[abc]`
//! - Any character: `.`
//! - Rule references: `rulename`
//! - Alternation: `|`
//! - Grouping: `( ... )`
//! - Repetition: `*`, `+`, `?`, `{n}`, `{n,m}`, `{n,}`
//! - Comments: `# ...`
//! - Unicode escapes: `\xNN`, `\uNNNN`, `\UNNNNNNNN`
pub use GrammarError;
pub use ;
pub use ;