Crate pipecrab_lm_llamacpp 

Native llama.cpp implementation of Pipecrab’s language-model capability.

LlamaCpp is a lightweight, cloneable handle. A dedicated worker thread owns llama.cpp’s model and context for their entire lifetime, streams decoded text as ModelDelta::Text through Pipecrab’s ModelStream, and checks an atomic cancellation flag between tokens. The worker arrangement keeps native inference state off the pipeline thread and makes barge-in bounded by one decode step.

Tool calling

llama-cpp-2 exposes only the legacy role/content chat-template call, so a ToolDialect owns every model-specific string: it renders declarations into the system message, converts the tool schemas to a GBNF, and reads a call back out of the token stream. The default is ChatMlXml.

The grammar is attached lazily, triggered by the dialect’s open delimiter: until a call begins, sampling is unconstrained and an ordinary reply streams as it would with no tools at all. Once a call begins, its text is withheld from the stream and surfaces as one ModelDelta::ToolCall — a fragment that escaped would be spoken aloud by the TTS stage. Because the grammar’s entry rule is one call, a turn yields at most one, and no text follows it.

Passing no tools disables all of it: the prompt and the delta stream are exactly what a dialect-free adapter produces.

The crate builds llama.cpp with Metal on iOS and the optimized ARM CPU backend on Android. Android uses the shared C++ runtime expected by an NDK application; callers can request GPU layers in LlamaCppConfig, but should only do so after a device-specific startup smoke test.

Structs

ChatMlXml

<tool_call>{"name": …, "arguments": …}</tool_call>, with declarations in a <tools> block.

LlamaCpp

Cloneable handle to a dedicated native llama.cpp inference worker.

LlamaCppConfig

Construction settings for a native llama.cpp worker.

Enums

LlamaCppBuildError

Why a llama.cpp worker could not be constructed.

Constants

TOOL_CALL_ROOT

Name of the GBNF entry rule a ToolDialect::grammar must define.

Traits

ToolDialect

One model family’s tool-calling convention.