pipecrab_lm_llamacpp/
lib.rs

1//! Native llama.cpp implementation of Pipecrab's language-model capability.
2//!
3//! [`LlamaCpp`] is a lightweight, cloneable handle. A dedicated worker thread
4//! owns llama.cpp's model and context for their entire lifetime, streams decoded
5//! text as [`ModelDelta::Text`](pipecrab_lm::ModelDelta::Text) through Pipecrab's
6//! [`ModelStream`](pipecrab_lm::ModelStream), and checks an atomic cancellation
7//! flag between tokens. The worker arrangement keeps native inference state off
8//! the pipeline thread and makes barge-in bounded by one decode step.
9//!
10//! # Tool calling
11//!
12//! `llama-cpp-2` exposes only the legacy role/content chat-template call, so a
13//! [`ToolDialect`] owns every model-specific string: it renders declarations
14//! into the system message, converts the tool schemas to a GBNF, and reads a
15//! call back out of the token stream. The default is [`ChatMlXml`].
16//!
17//! The grammar is attached *lazily*, triggered by the dialect's open delimiter:
18//! until a call begins, sampling is unconstrained and an ordinary reply streams
19//! as it would with no tools at all. Once a call begins, its text is withheld
20//! from the stream and surfaces as one
21//! [`ModelDelta::ToolCall`](pipecrab_lm::ModelDelta::ToolCall) — a fragment that
22//! escaped would be spoken aloud by the TTS stage. Because the grammar's entry
23//! rule is one call, a turn yields at most one, and no text follows it.
24//!
25//! Passing no tools disables all of it: the prompt and the delta stream are
26//! exactly what a dialect-free adapter produces.
27//!
28//! The crate builds llama.cpp with Metal on iOS and the optimized ARM CPU backend
29//! on Android. Android uses the shared C++ runtime expected by an NDK application;
30//! callers can request GPU layers in [`LlamaCppConfig`], but should only do so
31//! after a device-specific startup smoke test.
32#![forbid(unsafe_code)]
33#![warn(missing_docs)]
34
35mod config;
36mod dialect;
37mod intercept;
38mod worker;
39
40pub use config::{LlamaCppBuildError, LlamaCppConfig};
41pub use dialect::{ChatMlXml, TOOL_CALL_ROOT, ToolDialect};
42pub use worker::LlamaCpp;
pipecrab_lm_llamacpp/lib.rs

pipecrab_lm_llamacpp/
lib.rs
