Skip to main content

Crate llmman

Crate llmman 

Source

Modules§

auth
API keys and TLS trust for talking to llmman serve — the client half; enforcement is cmd::serve::auth.
chat_template
What a model’s chat template lets a request choose about thinking, read off the template text: the reasoning_effort levels it compares against, whether it reads enable_thinking, whether it thinks at all. Qwen3.8 takes low/medium/xhigh and raises on anything else; gpt-oss reads reasoning_effort but names only its default; Qwen3.5 and Gemma 4 have just the switch; Llama 3 has none. llmman launch offers an integration exactly these (see cmd::launch::opencode::opencode_variants), never a level the model rejects. Also whether it renders a request’s tools (handles_tools).
cmd
config
llmman.conf — one config file, read once, shared by every consumer.
container
Runs llama-server (or, for a safetensors model, vllm serve) inside a container (Linux only, via --runtime docker|podman, and the first thing the default --runtime auto tries) instead of as a local process, auto-selecting the matching ghcr.io/ggml-org/llama.cpp:server-<backend> image for whatever GPU acceleration the host actually has — see https://github.com/ggml-org/llama.cpp/blob/master/docs/docker.md for the full image list and their docker run flags, which [GpuBackend::engine_args] mirrors for the subset detected here. The same host probe picks the vLLM image (vllm/vllm-openai, rocm/vllm or vllm/vllm-openai-cpu, per architecture) — see [VllmBackend] — and SGLang’s lmsysorg/sglang ([VllmBackend::sglang_image_ref]).
daemon
Shared client-side helpers for talking to a local llmman serve instance over its Ollama-protocol HTTP API — http://127.0.0.1:17434 by default, overridable via LLMMAN_HOST.
ffi
Safe wrappers around the CGO-exported Go shim functions.
fmt
Small display-formatting helpers shared by CLI subcommands that print a table of images/models (list, ps) — kept in one place rather than duplicated per-file so the two commands’ NAME/ID/SIZE columns always render identically for the same underlying digest/byte count.
fsutil
Whole-file writes a reader never sees half done, writes only their owner can read, the free space a download needs, and the SHA-256 of a file or of what is written through a writer.
gguf
Minimal GGUF (https://github.com/ggml-org/ggml/blob/master/docs/gguf.md) header reader — reads just the metadata key/value section and the tensor-info array (name/shape/type — never the tensor data itself), which is all cmd::show needs to report a GGUF model’s architecture, parameter count, quantization, context length, and chat template without extracting/loading the whole file.
harmony
Parser for OpenAI’s “Harmony” response format — the <|start|>...<|channel|>...<|message|>...<|end|> channel-based format used by gpt-oss-style models to interleave a final (user-visible), analysis (reasoning/thinking), and commentary (tool-call) channel in one raw token stream.
hf
HuggingFace Hub support: auth, reference classification, and (in api/client/download/oci/pull/transfer/progress) the full pull/transfer fetch path, run natively and using hf-xet directly for Xet-backed files (see crate::xet_fetch).
hostgpu
Cross-platform detection of the GPU/accelerator, if any, available on the local host — used by crate::llama_release to pick which prebuilt llama-server release asset to download for this machine, and by crate::container to pick which ghcr.io/ggml-org/llama.cpp (or, for a safetensors model, vLLM) container image to run instead (--runtime docker|podman) — all share this one probe rather than detecting the host twice.
hybrid
Hybrid model pairs: one model name carrying a local model and a hosted one, with llmman serve choosing which serves each request.
imagegen
llmman run for media generation models, after ollama’s (since removed) x/imagegen/cli.go: one prompt in, one file out (png, or mp4 / wav with --video / --audio), saved next to the shell and shown inline where the terminal can. Progress comes as server-sent events.
llama_release
Downloads and caches a prebuilt llama-server from ggml-org/llama.cpp’s own GitHub Releases, picking the release asset that matches this host’s OS/arch and (per crate::hostgpu::detect) its best available GPU backend — the same “auto-detect and fetch the matching accelerated runner” step Ollama’s installer/runtime does for its own bundled llama-server builds (see Ollama’s discover/ package and scripts/install.sh’s check_gpu), applied here to llmman’s PATH-optional llama-server dependency instead.
mediagen
Image, video and audio generation with diffusion models on top of the ggml/llama shared libraries of a llama.cpp release, loaded at runtime.
metrics
Prometheus text-format metrics for llmman serve — the data behind its GET /metrics route (see cmd::serve::handle_metrics), which only exists when LLMMAN_METRICS=1.
mlx_release
Installs mlx-lm’s mlx_lm.server on macOS the way crate::llama_release fetches llama-server, in [crate::managed]’s shape:
modelpack
Reusable “resolve a local OCI-store reference to servable model files” logic — the CNCF ModelPack (https://github.com/modelpack/model-spec) equivalent of huggingface_hub.snapshot_download.
oauth
Docker Hub’s web-based device-authorization login flow — the “USING WEB-BASED LOGIN” UX docker login shows by default — ported from docker/cli’s internal/oauth package (manager.LoginDevice, api.GetDeviceCode/WaitForDeviceToken/GetAutoPAT).
ordered
A JSON object that keeps its members in order, which serde_json’s own map does not. /v1/systemone needs it end to end: questions are read in the order written and answers come back in it.
promptlog
The prompt log: llmman serve appends one JSON line per generation request to prompts.jsonl beside the store; llmman log reads it.
providers
The models.dev provider catalog, fetched at runtime rather than baked into this binary, so a provider models.dev gains tomorrow works without an llmman release.
shortnames
Short-name alias resolution — from the [aliases] table of llmman.conf, at runtime. Beyond [OLLAMA_NAMES], nothing is compiled into the binary.
sources
Model sources that are neither an OCI registry nor the HuggingFace Hub. Ported from the Go shim’s uri_sources.go, whose deletion also removed the last caller of hf.go.
speech
Speech recognition on the ggml library of the pinned llama.cpp release. model reads the Canary GGUF description, checks its tensor contract and uploads the weights onto a mediagen::backend::Backend. wav and mel turn PCM16 WAV input into the log-mel features the encoder reads. encoder builds the FastConformer encoder as a ggml graph on that backend, with the host-side arithmetic it consumes: subsampled frame count, relative-position table and the folded BatchNorm. prompt builds the canary2 decoder prompt from the hyperparameters and the vocabulary; decoder runs the Transformer decoder over the encoder output with a KV cache; Vocab::detokenize turns decoder output back into text. server is the worker process’s HTTP side: it parses and bounds a transcription request, runs that sequence one request at a time and answers in the OpenAI shapes.
storage
thinking
Native <think>...</think> extraction — a direct port of ollama’s thinking/parser.go, used as a fallback when the inference backend doesn’t already separate reasoning from content itself (see cmd::serve::stream’s own oai_chunk_to_content, which prefers a backend’s structured reasoning_content/thinking delta field whenever one is present and only ever falls back to this parser scanning raw content text for literal tag characters).
tk
llmman tk <command> [args...]: runs a command and prints its output compacted, so a coding agent spends fewer tokens on noise.
usage
The usage ledger: llmman serve appends a line per reply that reports token usage to usage.jsonl beside the store; llmman usage reads it.
verify
Signature trust policy: which references must be signed, and by whom.
webui
The web UI’s files, gzipped into the binary by build.rs as the ASSETS table: (path, content-type, etag, bytes), keyed by the path relative to webui/. Served by cmd::serve::webui.
xet_fetch
Real Xet-protocol downloads for HuggingFace files, via hf-xet (huggingface/xet-core, Apache-2.0). huggingface_hub’s own client refuses to fetch any file over 50GB through a plain HTTP GET at all (file_download.py: “Install hf_xet … for xet-powered downloads.”), requiring this protocol instead.

Macros§

debug_log
Prints a [llmman] [debug] ... line to stderr, only when debug_enabled. A macro (not a plain function) so the format! args aren’t even evaluated when debug logging is off.

Functions§

data_root
~/.local/share/llmman (%LOCALAPPDATA%\llmman on Windows): the parent of the default store and of everything else llmman keeps on disk for itself — the llama.cpp release (llama_release), the uv binary and mlx-lm environment (mlx_release), each in the shape managed describes. Not affected by LLMMAN_MODELS, which relocates only the store.
debug_enabled
Whether verbose diagnostic logging is enabled, from LLMMAN_DEBUG (mirrors Ollama’s OLLAMA_DEBUG). llmman has only one verbosity tier, so any truthy value or non-zero integer enables it (Ollama’s OLLAMA_DEBUG=2 TRACE spelling also works, just maps to the same tier).
default_cache
Path to the extracted-model cache — a sibling of the store (<store>/../cache), matching what llmman serve (the long-running server that extracts most models) uses. The single canonical answer for “where is the cache”, so extraction and the GC sweep agree on it rather than one command writing to <store>/cache and another cleaning <store>/../cache.
default_store
Path to the local OCI store, from LLMMAN_MODELS (mirrors Ollama’s OLLAMA_MODELS) or else ~/.local/share/llmman/store (%LOCALAPPDATA%\llmman\store on Windows).
env_flag_set
Whether an opt-out variable (LLMMAN_NOPRUNE, LLMMAN_NOHISTORY) is set: anything but unset, blank, or 0/false/no/off.
find_on_path
The first name (name.exe on Windows) that is a file in a PATH directory.