Crate drama_llama

Expand description

§`drama_llama`

drama_llama runs language models on your own hardware behind an API shaped like Anthropic’s Messages API. It speaks misanthropic’s Prompt, Message and Block types directly — not a lookalike, the same types — so code written against the Anthropic API drives a local GGUF by swapping the transport and nothing else.

It is a work in progress and not intended for production use. The API will change.

The part worth your attention is what happens to structured output. When you ask a hosted API for JSON matching a schema, you are asking politely. Here the schema is compiled to a GBNF grammar and enforced inside the sampler, one token at a time: tokens that would break the schema are removed from the distribution before a choice is made. Malformed JSON is not unlikely, it is unreachable.

// Compiled and type-checked by CI — not run, since it wants weights. The
// cfg gate keeps the doctest building when these features are off.
#[cfg(all(feature = "llama-cpp", feature = "json-schema"))]
fn main() -> Result<(), Box<dyn std::error::Error>> {
    use drama_llama::{
        FromPath, LlamaCppOptions, LlamaCppSession, Prompt, Role,
    };
    use schemars::JsonSchema;
    use serde::{Deserialize, Serialize};

    /// Field order is generation order. `summary` is written first, so
    /// the model has already said what the bug *is* before it has to
    /// commit to a severity — each field is context for the next.
    #[derive(Debug, Serialize, Deserialize, JsonSchema)]
    struct Triage {
        /// One-line, imperative summary of the underlying problem.
        summary: String,
        /// How bad it is, chosen after summarizing.
        severity: Severity,
        /// Concrete, ordered steps to reproduce.
        repro_steps: Vec<String>,
        /// True when the report says the behavior regressed.
        is_regression: bool,
    }

    #[derive(Debug, Serialize, Deserialize, JsonSchema)]
    #[serde(rename_all = "snake_case")]
    #[schemars(rename_all = "snake_case")]
    enum Severity {
        Low,
        Medium,
        High,
        Critical,
    }

    let mut session = LlamaCppSession::from_path_with(
        "models/model.gguf".into(),
        LlamaCppOptions::default().with_n_ctx(8192),
    )?;

    let prompt = Prompt::default()
        .system("You triage incoming bug reports.")
        // A worked exemplar teaches field *depth* — that `repro_steps`
        // should be concrete and non-empty — which a bare schema cannot
        // express. It seeds the schema too, so the two cannot drift apart.
        .add_examples([(
            "Login does nothing in Safari. Started after last week's release.",
            Triage {
                summary: "Login button unresponsive on Safari".into(),
                severity: Severity::High,
                repro_steps: vec![
                    "Open the app in Safari".into(),
                    "Click 'Log in'; observe no network request".into(),
                ],
                is_regression: true,
            },
        )])?
        .add_message((Role::User, "Checkout total shows $0.00 on mobile."))?;

    // The response has the Anthropic shape — content, usage, stop
    // reason — and `.json()` parses its text block, skipping any
    // leading thought blocks. The parse cannot fail on malformed JSON:
    // the model was not able to emit any.
    let triage: Triage = session.complete_response(&prompt)?.json()?;
    println!("{triage:#?}");
    Ok(())
}

#[cfg(not(all(feature = "llama-cpp", feature = "json-schema")))]
fn main() {}

The grammar engine underneath is ours — a pure-Rust GBNF parser, matcher and lazily-built DFA cache, not a call into llama.cpp’s. So it is usable on its own, with no backend and no C dependency at all:

use drama_llama::{GrammarState, SamplingMode};

let gbnf = r#"
    root ::= "{" ws "\"ok\"" ws ":" ws bool ws "}"
    bool ::= "true" | "false"
    ws   ::= [ \t\n]*
"#;

// As a sampling mode this constrains generation token by token.
let mode = SamplingMode::grammar(gbnf).unwrap();
assert!(matches!(mode, SamplingMode::Grammar(_)));

// The same grammar, driven by hand. `completes_with` asks whether the
// bytes are accepted *and* land in a final state; `accepts_bytes` asks
// only whether they are a legal prefix. Neither mutates the matcher.
let state = GrammarState::from_source(gbnf).unwrap();
assert!(state.completes_with(br#"{"ok": true}"#));
assert!(state.accepts_bytes(br#"{"ok": "#));
assert!(!state.accepts_bytes(br#"{"ok": maybe"#));

§The layers

You can enter at whichever level you need. Each is a thin, public wrapper over the one below it.

Layer	Type	What it gives you
5	`SessionTransport` / `LocalTransport`	Implements `misanthropic::Transport`, so `Chat` loops and agent reactors written for the API drive a local model unchanged.
4	`Session<B>`	The chat-shaped API: `complete`, `complete_text`, `complete_blocks`, `complete_stream`, `complete_response`. Owns templating, tool dialects, the prefix cache, and grammar resolution.
3	`Predictor` family	`predict_candidates`, `predict_tokens`, `predict_pieces`, `predict` — iterators. `CandidatePredictor::record_choice` lets you pick the token, which is how forced-continuation scoring works.
2	`Engine<B>`	Decoder + model + optional vision, plus direct KV-cache control (`memory_seq_rm`, `checkpoint_pos`, `restore_to`, …).
1	`Candidates`, `SamplerConfig`	Every sampling method, translated to Rust. No calls into `llama.cpp`’s sampler chain.
0	`backend`	`Backend`, `Decoder`, `Model`, `Vision` traits. Compiles with `--no-default-features`: no C dependency.

§Supported features

	Feature flag
Structured output	`json-schema`	A `schemars`-derived type becomes a sampling grammar. Optional `<think>…</think>` preamble, phase-split so the thought runs unconstrained at full speed.
Tool calling	(always on)	Per-model dialects derived by analyzing each model’s own chat template, driving both the grammar emitter and the response parser. `ToolChoice::method` is guaranteed locally, not requested. Validated for Qwen 3.5/3.6, Gemma 4, gpt-oss (Harmony).
GBNF grammars	(always on)	Pure-Rust parser, matcher and lazy-DFA cache. Sampling checks the one sampled token first and only falls back to an O(vocab) mask on rejection.
Prefix caching	(always on, opt-in at runtime)	Multi-slot, breakpoint-driven, LRU with TTL. Honors Anthropic `cache_control` ephemeral markers. One slot per agent, so an N-agent workload caches N prefixes instead of thrashing one.
Chat templates	(always on)	The model’s own Jinja `tokenizer.chat_template`, rendered by `minijinja`. No per-model prompt formats hardcoded here.
Images	`media`, `mtmd`	`media` is pure Rust (decode via the `image` crate, never `mtmd`’s bundled `stb_image`); `mtmd` adds llama.cpp’s multimodal backend. Images render out-of-band through a per-call random sentinel — the projector never sees prompt text.
Sampling	(always on)	Greedy, temperature, top-k, top-p, min-p, tail-free, locally typical, Mirostat v1/v2, plus `SplitP`/`SplitL`/`Deny` which have no llama.cpp counterpart. Chained: each mode narrows the candidate set.
Repetition penalties	(always on)	N-gram based, windowed and decaying, with category exclusions so common English or JSON punctuation isn’t penalized. Region-aware inside grammar free-text spans.
HTTP server	`axum`	`blallama` — an Anthropic-compatible `/v1/messages` server over a local model, with an SSE `/probe` channel.
Accelerators	`cuda`, `cuda_f16`	Metal is automatic on macOS.
Async	`tokio`	`SessionTransport`, `FromPath::from_path_async`.
Sidecars	`toml`	Per-model `sampling.toml` / `dialect.toml` / template files beside the GGUF.

§Backends

Two, behind one Backend trait:

llama-cpp (default) — llama.cpp via llama-cpp-sys-3. CUDA and Metal.
moeflux — a Metal-native streaming-MoE runtime, macOS only. Selects its model at compile time: exactly one of moeflux-model-qwen3-6-35b-a3b, moeflux-model-qwen3-5-a17b, or moeflux-model-cogito-v2-671b. The last is ~336 GB at 4-bit and streams experts from SSD, which is how a 671B model runs on a 96 GB laptop.

Both can be linked at once. When they are, name the alias (LlamaCppSession) rather than a bare Session — a bare Session::from_path only infers a backend when exactly one exists.

§Examples

Eighteen of them in examples/. Each carries a module doc explaining not just what it does but why it is shaped that way. The ones worth reading first:

Example
`strawberry`	Typed tool use with the `#[tool]` macro. Locally, `ToolChoice::method` compiles to a grammar, so the call is guaranteed.
`whodunit`	Structured output into a typed `CaseFile`, streamed block by block so thoughts arrive as they parse.
`prompt_caching`	The prefix cache, demonstrated self-referentially: the system prompt embeds this README and the transport’s own source, then asks about them.
`swarm`	Five agents, one GPU, a `#[tool]`-built mail system and a postage ledger. One cache slot per seat.
`whoami`	The raw `Engine` + `CandidatePredictor` layer: scores candidate model names by forced continuation, reading the distribution instead of the string.
`unhelpful`	Steering by prefilling the model’s own reasoning with an unclosed thought block.

just example whodunit
cargo run --release --example strawberry --features "tokio,cli,json-schema"

There are also three binaries: blallama (the HTTP server), regurgitater (tests local models for memorized content), and settings_tool (an egui sampler-settings editor).

§Testing

605 tests across 27 binaries in the default configuration — 486 that run in seconds and 119 that load real weights onto a real accelerator. The model-backed tier is #[ignore]d so the fast loop stays fast, and the whole topology — which features × which tests — lives in one place, scripts/test.py. The justfile delegates to that script, the git hooks call the justfile, and CI calls the script directly, so the tests that gate a commit are byte-for-byte the ones that gate a push.

(That is cargo nextest list’s count for the llama-cpp configuration, not a grep for #[test]. The two disagree, and only one of them is what runs.)

just setup            # cargo-nextest + cargo-llvm-cov (once)
just install-hooks    # point git at .githooks/ (once)

just test             # the fast tier: no weights, fully parallel
just test ignored     # ONLY the model tests, serialized
just test all         # everything
just test moeflux     # the moeflux configuration, plus cross-backend
just test NAME        # anything matching NAME, any tier, uncaptured

just check            # rustfmt + rustdoc, what the pre-commit hook runs
just permutations     # every feature configuration compiles, test targets too
just doctest          # the doctests, including the ones on this page
just coverage         # instrumented run + report

Everything goes through cargo-nextest, which gives each test its own process. Do not use plain cargo test: it overlaps test binaries, which --test-threads=1 does not fix, so two 19 GB models load at once and the OOM surfaces as a decode failure that reads like a regression.

Run python3 scripts/test.py --help for the real interface — and use it directly on Windows, since the recipe bodies are bash.

§Roadmap

Automatic batch scheduling and better parallelism
Runtime model-variant selection for moeflux, replacing the compile-time feature selection
Stream misanthropic::stream::Event from Session::complete_stream (#26)
Tokenization in the browser
Backends beyond llama.cpp and moeflux — an NPU target is the long-term goal

See CHANGELOG.md for what has already landed, and the issue tracker for what is actively broken.

§Known issues

A KV-dirty llama_decode failure leaves the cache unreconciled (#52).
A context-full stop is reported as a grammar violation (#36).
moeflux’s memory_seq_cp / memory_seq_keep silently no-op and report success (#42).

§Contributing

Code is poetry. Make it pretty.
Respect is universal.
Use rustfmt — just install-hooks makes that automatic.

§Generative AI Disclosure

Generative AI, specifically Microsoft’s Bing Copilot, GitHub Copilot, and Dall-E 3 were used for portions of this project. See inline comments for sections where generative AI was used. Completion was also used for getters, setters, and some tests. Logos were generated with Dall-E and post processed in Inkscape.
Anthropic’s Claude (primarily as Claude Code) is a direct collaborator on this project and co-authors commits where it contributed. git log is the authoritative record — grep for Co-Authored-By: Claude — and CONTRIBUTORS.md summarizes the surface areas. As of v0.8.0 those include the llama.cpp API migration, the sampling-mode suite (JSON, GBNF, tool-choice, structured output), the Jinja chat- template renderer, the prompt-caching layer, the grammar matcher performance finish line (lazy-DFA cache + thought/JSON phase-split), and the Backend split that lets the same Session/Engine surface drive either llama.cpp or moeflux’s Metal MoE runtime.

Re-exports§

pub use data::IgnoreCategory;
pub use backend::ImageDecodeError;
pub use backend::Backend;
pub use backend::Decoder;
pub use backend::Image;
pub use backend::ImageInfo;
pub use backend::ImageNewError;
pub use backend::LogLevel;
pub use backend::MediaChunk;
pub use backend::MediaSpan;
pub use backend::Model;
pub use backend::NoVision;
pub use backend::NotImplemented;
pub use backend::Token;
pub use backend::TokenData;
pub use backend::Vision;
pub use prompt::Tool;
pub use output_config::compile_output_config;
pub use output_config::compile_prompt_output_config;
pub use output_config::grammar_for_output_config;
pub use output_config::CompiledOutputConfig;
pub use output_config::OutputConfigError;
pub use output_config::OutputConfigOptions;
pub use dialect::CallSyntax;
pub use log::clear_log_callback;
pub use log::restore_default_logs;
pub use log::set_log_callback;
pub use log::silence_logs;
pub use sidecar::load_call_syntax;
pub use sidecar::load_sample_options;
pub use sidecar::seed_config_for;
pub use sidecar::write_call_syntax;
pub use sidecar::write_sample_options;
pub use sidecar::load_template_source;
pub use sidecar::SidecarError;
pub use minijinja;

Modules§

backend: Backend-agnostic primitives shared across decoder/model implementations.
cli: CLI-shaped types shared by this crate’s binaries and its examples.
data
dialect: Per-model tool-call dialects: CallSyntax and the template analyzer that derives it.
log: Process-global log callback for llama.cpp and ggml.
output_config: Prompt::output_config → SamplingMode compiler.
prompt: Chat prompt primitives, re-exported wholesale from misanthropic.
sidecar: Per-model sidecar files.

Structs§

BlockStream: Streaming Iterator over crate::Blocks, produced by Session::complete_stream. Yields each structured block (thought, tool call) as soon as its closing marker arrives; prose streams incrementally as it resolves.
CachedPrompt: A Prompt with an immutable cache prefix.
CandidatePredictor: An iterator that predicts a sequence of candidate distributions.
Candidates: A container for candidate tokens.
ChatTemplate: A compiled chat template tied to a specific model’s tokens.
CompiledGrammar: A compiled grammar plus its lazy-DFA cache — the config half of a grammar constraint. Immutable: matching position lives in the per-call sampler state (StackState), never here.
Content: Content of a Message, stored as a sequence of Blocks.
DeferredGrammar: A grammar that starts suspended and activates once a specific byte sequence appears in the predictor’s generated text. Activation is driven by TokenPredictor: when the trigger is found in the accumulated output, the matcher in SamplerState is flagged active and any bytes emitted after the trigger are fed into it so it lines up with the model’s byte position.
Engine: An Engine encompasses everything needed to run inferences. It bundles a crate::Decoder (context + KV cache) with a crate::Model (weights + tokenizer) via a single Backend parameter. Use the LlamaCppEngine / MoefluxEngine type aliases (feature-gated) for the common backends.
Grammar: A compiled GBNF grammar.
GrammarState: Active matching state for a Grammar.
GrammarStats: Cumulative statistics about grammar_filter calls since process start (or since the last grammar_stats_reset).
InvalidProbability: Error for invalid probability values.
JsonState: Pushdown-automaton state for JSON parsing at the byte level.
LlamaCppBackend: Tag for the llama-cpp Backend. Use as a type parameter for Engine or Session.
LlamaCppDecoder: llama.cpp-backed decoder: owns a llama_context, manages the KV cache, runs decode passes, and exposes logits / embeddings.
LlamaCppModel: An ergonomic wrapper for a llama.cpp model.
LlamaCppOptions: Load-time configuration for LlamaCppEngine and Session<LlamaCppBackend>.
Mtmd: A loaded multimodal projector (mmproj): llama.cpp’s mtmd_context.
MtmdParams: Construction parameters for Mtmd — the small, stable subset of mtmd_context_params we expose. Everything else stays upstream default; notably the media marker, which Session assumes is the default <__media__>.
NGram: An immutable N-gram of tokens.
NGramData: Metadata about an Ngram.
NGramStats: A map of NGram metadata.
PiecePredictor: A predictor that predicts pieces of text.
PredictOptions: Options for prediction.
Predicted: Contains a token and the associated piece. This is a convenience struct to avoid ackward iterator usage when both the token and piece are needed.
Predictor
PrefixCacheConfig: Configuration for the multi-slot prefix cache (Session::with_prefix_cache_config).
Probability: A Probability is a wrapper around a floating point number that represents a probability. It is guaranteed to be between 0.0 and 1.0.
ProbeCtx: Per-token state passed to ProbeHook::on_token.
Prompt: Request for the Anthropic Messages API.
RenderOptions: Options passed to ChatTemplate::render_with.
RenderedWithBreakpoints: Rendered prompt plus one partial render per cache_control breakpoint.
RepetitionOptions: Options for Candidates::penalize_repetition.
SamplerConfig: Options determining how raw logits are turned into a token. This is used by Candidates::sample_token and associated functions.
SamplerState: Everything a generation call mutates while sampling. See the module docs for the purity contract.
SamplingParams: The scalar sampling knobs that a model recommends and that the Anthropic wire format can carry — the common vocabulary shared by llama.cpp’s general.sampling.* GGUF metadata, OpenAI’s and Anthropic’s request bodies, and every --temp-style CLI on earth.
Session: Chat-style inference session: owns an Engine + ChatTemplate plus the builder-configured defaults for each complete_* call.
SessionTransport: A misanthropic::Transport over a locally-owned Session — see the module docs for the concurrency model.
Snapshot: A snapshot of the pre-everything candidate distribution, captured before the sampling-mode chain (rep penalty, deny mask, top-K / top-P / mirostat / grammar / …) consumes the candidates.
SnapshotOpts: Options controlling Candidates::capture_snapshot.
TokenDataArray
TokenPredictor
TokenTrace: One generated-position entry in a Session::top_k_trace dump.
ToolChoiceOptions: Options for grammar_for_tool_choice.
TopKEntry: One candidate row inside a TokenTrace.

Enums§

Block: A Content Block of a Message.
ChatTemplateError
DecodeError: Possible errors when calling LlamaCppDecoder::decode.
FlashAttention: Flash Attention policy for a new crate::LlamaCppEngine context.
GrammarError
JsonError
Mirostat: Which mirostat algorithm a model recommends, plus its parameters. Mirostat is terminal — it yields a single token — so it never composes with the truncation knobs in SamplingParams.
NewError: Possible errors when creating a new crate::Engine or LlamaCppDecoder.
PromptBreakpoint: A position in a Prompt where the caller placed a cache_control marker.
RepetitionError
Role: Role of the Message author.
SamplingMode
SessionError: Errors from Session.
Sorted: Sort state of the candidates.
ToolChoice: Constrain the Assistant’s choice of CustomMethodDefs.
ToolChoiceError: Errors from grammar_for_tool_choice.

Constants§

TOS

Traits§

FromPath: Load a Session from a path, with whatever load-time options its backend understands.
LocalTransport: A SessionTransport with its backend erased — what a caller holds when the model behind it is chosen at runtime rather than at compile time.
ProbeHook: Per-token observer for crate::TokenPredictor.

Functions§

apply_request_sampling: Fold a request’s sampling knobs into an existing mode chain.
deferred_grammar_for_prompt: Build the lazy (trigger-activated) tool-call constraint for a prompt whose tool_choice is Auto — or absent, which the Anthropic API treats as auto — with tools advertised.
gpu_device_names: Names of the GPU-class compute devices ggml discovered, in registry order. Empty means every model will run on the CPU.
grammar_for_prompt: Derive the tool-choice grammar directly from a Prompt. Reads prompt.tool_choice and prompt.tools; returns Ok(None) when the prompt imposes no constraint (no tool_choice, Auto, or no advertised tools).
grammar_for_tool_choice: Build a SamplingMode::Grammar that forces the model’s output to match the chosen tool-call shape.
grammar_stats_enabled: Whether DRAMA_LLAMA_GRAMMAR_STATS was set to a truthy value when first checked. Cached — subsequent env var changes are ignored.
grammar_stats_reset: Reset cumulative statistics. Useful to measure a single phase of generation in isolation.
grammar_stats_snapshot: Snapshot cumulative grammar_filter statistics. Returns zeros when collection is disabled.
llama_quantize: Quantize a Llama model.
tokenize_with_breakpoints: Tokenize the full render and each partial in rendered, returning the full token stream plus the sorted, deduplicated breakpoint token indices.

Type Aliases§

AssistantMessage: A message guaranteed to be from the assistant.
LlamaCppEngine: Convenience alias for the llama.cpp-backed pair. Use LlamaCppEngine::from_path(...) etc. when you want the default backend without turbofish.
LlamaCppSession: Convenience alias for the llama.cpp-backed session, parallel to crate::LlamaCppEngine.
Message: A message whose Role is only known at runtime.
UserMessage: A message guaranteed to be from the user.