Crate llama_crab 

llama-crab — safe, ergonomic Rust bindings to llama.cpp.

Quickstart

use llama_crab::{Llama, LlamaParams};

let mut llama = Llama::load(LlamaParams::new("model.gguf").with_n_ctx(2048))?;
let resp = llama.create_completion("Hello, world!", 64)?;
println!("{}", resp.text);

Loading from Hugging Face

Pass a Hugging Face repository id (e.g. TheBloke/Llama-2-7B-Chat-GGUF) directly to LlamaParams::new; the library will download the GGUF to the official HF cache and load it. For repos with multiple .gguf files, specify the filename via with_hf_filename:

#[cfg(feature = "hf-hub")]
{
use llama_crab::{Llama, LlamaParams};

let mut llama = Llama::load(
    LlamaParams::new("TheBloke/Llama-2-7B-Chat-GGUF")
        .with_hf_filename("llama-2-7b-chat.Q4_K_M.gguf")
        .with_n_ctx(2048),
)?;
}

The model is cached at ~/.cache/huggingface/hub (or $HF_HOME/hub if set). Set HF_TOKEN for gated repos. Requires the hf-hub cargo feature.

Re-exports

pub use crate::backend::LlamaBackend;

pub use crate::backend::NumaStrategy;

pub use crate::batch::BatchAddError;

pub use crate::batch::LlamaBatch;

pub use crate::chat::Role;

pub use crate::context::LlamaContext;

pub use crate::context::LlamaContextParams;

pub use crate::error::LlamaError;

pub use crate::error::Result;

pub use crate::hf::HfDownloader;

pub use crate::hf::HfRepo;

pub use crate::hf::MockHfDownloader;

pub use crate::high_level::chat_completion::ChatMessage;

pub use crate::high_level::completion::Completion;

pub use crate::high_level::completion::CompletionChunk;

pub use crate::high_level::completion::CompletionLogprobs;

pub use crate::high_level::completion::CompletionOptions;

pub use crate::high_level::completion::StopReason;

pub use crate::high_level::completion::StreamControl;

pub use crate::high_level::completion::TokenLogprob;

pub use crate::high_level::tokenizer::FimTokens;

pub use crate::high_level::tokenizer::LlamaTokenizer;

pub use crate::high_level::tokenizer::Tokenizer;

pub use crate::high_level::Llama;

pub use crate::high_level::LlamaParams;

pub use crate::high_level::MobilePreset;

pub use crate::log::send_logs_to_tracing;

pub use crate::log::LogOptions;

pub use crate::logit_bias::LlamaLogitBias;

pub use crate::model::params::LlamaModelParams;

pub use crate::model::LlamaModel;

pub use crate::sampling::LlamaSampler;

pub use crate::sampling::SamplerChain;

pub use crate::token::LlamaToken;

pub use crate::token_data::LlamaTokenData;

pub use crate::token_data::LlamaTokenDataArray;

Modules

backend

Global backend initialization, NUMA strategy, device enumeration.

batch

Reusable batching primitive.

cache

Caching for previously-computed prompt prefixes.

chat

Chat message types, templates and tool calling.

context

LlamaContext and its parameters.

error

Error types for llama-crab.

hf

Hugging Face integration. Real network access requires the hf-hub feature.

high_level

High-level orchestrator: load model, create context, generate tokens.

json_schema

JSON Schema → GBNF grammar converter.

log

Forwarding llama.cpp/ggml logs into the tracing ecosystem.

logit_bias

LlamaLogitBias — a single (token, bias) pair, used by the logit-bias sampler.

model

LlamaModel and its parameters.

multimodalmtmd

Multimodal (vision + audio) support via mtmd.

sampling

Sampling strategies and the LlamaSampler wrapper.

speculative

Speculative decoding: a draft model proposes candidate tokens, the main model validates them in a single forward pass.

token

LlamaToken newtype and LlamaTokenAttr bitflags.

token_data

LlamaTokenData and LlamaTokenDataArray.

util

Small utilities used across the crate.