llama-cpp-sys-4

Raw bindgen-generated bindings to llama.cpp, plus the C/C++ build logic that compiles the library.

llama.cpp version: 0adcc3bb5 (b10502, incl. v0.1.2) · Crate version: 0.6.1

Unless you need access to a symbol not yet exposed by llama-cpp-4, use that crate instead — it provides a safe API over these raw bindings.

For application code, start with:

use llama_cpp_4::prelude::*;

What's included

• llama_* functions and types from llama.h
• ggml_* functions and types from ggml/include/ggml.h
• LLAMA_* constants
• common_tokenize and common_token_to_piece from common/common.h
• The entire llama.cpp static library (or shared, with dynamic-link)

Feature flags

dflash2 — vendored pre-merge patch

--features dflash2 applies patches/0006-dflash2.patch, cut from upstream PR #27342, which adds DFlash2 drafting (grouped dynamic depthwise convolution + candidate selector). It is opt-in and off by default because that PR has not been merged upstream.

What that means in practice:

• The patch adds new GGUF KV keys (dflash.conv_kernel_size, dflash.selector_rank, …) and tensors under the existing LLM_ARCH_DFLASH architecture. With the feature on, this build recognises DFlash2 checkpoints that stock llama.cpp releases do not.
• Only the C++ needed by the library is vendored — common/ and src/. The PR's Python side (gguf-py/, conversion/qwen.py) is not included, so this crate can run a DFlash2 checkpoint but cannot convert one; use the PR branch's convert_hf_to_gguf.py for that.
• The patch is staged after 0003–0005, which also touch common/speculative.cpp.
• Upstream may change the PR before it merges. When it does merge, this patch should be dropped and the feature turned into a no-op or removed.

DFlash2 checkpoints are detected from GGUF metadata, so no separate speculative type is needed — select a DFlash draft via Eagle3Session::new_dflash in llama-cpp-4.

Prebuilt libraries

The exact speculative-state, decode-lifecycle, and fail-closed EAGLE-3 process patches require native libraries built from the same patched source. No published release archive carries a patch-identity envelope yet, so prebuilt currently emits a warning and uses the verified source build. LLAMA_PREBUILT_DIR fails closed while those patches are active rather than mixing patched headers with unverifiable native objects.

Once release assets carry an exact patch identity, the intended interface is:

# Automatic download + cache, when a compatible release asset is available
cargo build -p llama-cpp-sys-4 --features prebuilt

# Prefetch manually, then build
./scripts/fetch-prebuilt.sh
export LLAMA_PREBUILT_DIR=target/llama-prebuilt-cache/0.6.1/llama-prebuilt-...
cargo build -p llama-cpp-sys-4

Release assets are named llama-prebuilt-{linux|macos|windows}-{target}-{cpu|vulkan|blas|metal}-{static|dynamic}.tar.gz and published by .github/workflows/prebuilt-llama.yml on version tags.

cuda, hip, webgpu, opencl, and q1 have no published prebuilts yet — those builds always compile locally.

Building

The crate compiles llama.cpp from the vendored submodule at build time using cc + cmake-style flags. No external llama.cpp installation is required.

# CPU only (default)
cargo build -p llama-cpp-sys-4

# Metal (macOS)
cargo build -p llama-cpp-sys-4 --features metal

# CUDA
cargo build -p llama-cpp-sys-4 --features cuda

# OpenMPI (distributed inference)
brew install open-mpi   # or apt install libopenmpi-dev
cargo build -p llama-cpp-sys-4 --features mpi

Build dependencies

• clang — required by bindgen to parse the C++ headers
• A C++17 compiler (GCC 9+, Clang 10+, MSVC 2019+)
• cmake and a supported CMake generator (Ninja is preferred)

Regenerating bindings

Bindings are regenerated automatically whenever build.rs or wrapper.h changes. The allowlist covers llama_*, ggml_*, LLAMA_*, and the two common_* functions.

# Force a full rebuild including binding regeneration
touch llama-cpp-sys-4/wrapper.h
cargo build -p llama-cpp-sys-4

Notable API changes (b4689 → b8249)

These are the upstream llama.cpp breaks handled in this crate:

Bindgen configuration

Key decisions in build.rs:

• derive_partialeq(true) with no_partialeq(...) overrides for structs containing function-pointer fields (avoids the unpredictable_function_pointer_comparisons lint).
• opaque_type("std::.*") — C++ STL types are opaque pointers.
• OpenMP auto-detection — reads GGML_OPENMP_ENABLED from the CMake cache rather than relying solely on the openmp feature flag, because some ARM toolchains enable OpenMP unconditionally.