What a model’s chat template lets a request choose about thinking,
read off the template text: the reasoning_effort levels it compares
against, whether it reads enable_thinking, whether it thinks at
all. Qwen3.8 takes low/medium/xhigh and raises on anything
else; gpt-oss reads reasoning_effort but names only its default;
Qwen3.5 and Gemma 4 have just the switch; Llama 3 has none.
llmman launch offers an integration exactly these (see
cmd::launch::opencode::opencode_variants), never a level the model rejects.
Also whether it renders a request’s tools (handles_tools).
Runs llama-server (or, for a safetensors model, vllm serve) inside
a container (Linux only, via --runtime docker|podman, and the first
thing the default --runtime auto tries) instead of as a local
process, auto-selecting the matching
ghcr.io/ggml-org/llama.cpp:server-<backend> image for whatever GPU
acceleration the host actually has — see
https://github.com/ggml-org/llama.cpp/blob/master/docs/docker.md for
the full image list and their docker run flags, which
[GpuBackend::engine_args] mirrors for the subset detected here. The
same host probe picks the vLLM image (vllm/vllm-openai, rocm/vllm
or vllm/vllm-openai-cpu, per architecture) — see [VllmBackend] —
and SGLang’s lmsysorg/sglang ([VllmBackend::sglang_image_ref]).
Shared client-side helpers for talking to a local llmman serve
instance over its Ollama-protocol HTTP API — http://127.0.0.1:17434
by default, overridable via LLMMAN_HOST.
Small display-formatting helpers shared by CLI subcommands that print a
table of images/models (list, ps) — kept in one place rather than
duplicated per-file so the two commands’ NAME/ID/SIZE columns always
render identically for the same underlying digest/byte count.
Whole-file writes a reader never sees half done, writes only their
owner can read, the free space a download needs, and the SHA-256 of a
file or of what is written through a writer.
Minimal GGUF (https://github.com/ggml-org/ggml/blob/master/docs/gguf.md)
header reader — reads just the metadata key/value section and the
tensor-info array (name/shape/type — never the tensor data itself),
which is all cmd::show needs to report a GGUF model’s architecture,
parameter count, quantization, context length, and chat template
without extracting/loading the whole file.
Parser for OpenAI’s “Harmony” response format — the
<|start|>...<|channel|>...<|message|>...<|end|> channel-based format
used by gpt-oss-style models to interleave a final (user-visible),
analysis (reasoning/thinking), and commentary (tool-call) channel
in one raw token stream.
HuggingFace Hub support: auth, reference classification, and (in
api/client/download/oci/pull/transfer/progress) the
full pull/transfer fetch path, run natively and using hf-xet
directly for Xet-backed files (see crate::xet_fetch).
Cross-platform detection of the GPU/accelerator, if any, available on
the local host — used by crate::llama_release to pick which
prebuilt llama-server release asset to download for this machine,
and by crate::container to pick which ghcr.io/ggml-org/llama.cpp
(or, for a safetensors model, vLLM) container image to run instead
(--runtime docker|podman) — all share this one probe rather than detecting the host
twice.
llmman run for media generation models, after ollama’s (since removed)
x/imagegen/cli.go: one prompt in, one file out (png, or mp4 / wav
with --video / --audio), saved next to the shell and shown inline
where the terminal can. Progress comes as server-sent events.
Downloads and caches a prebuilt llama-server from
ggml-org/llama.cpp’s own
GitHub Releases, picking the release asset that matches this host’s
OS/arch and (per crate::hostgpu::detect) its best available GPU
backend — the same “auto-detect and fetch the matching accelerated
runner” step Ollama’s installer/runtime does for its own bundled
llama-server builds (see Ollama’s discover/ package and
scripts/install.sh’s check_gpu), applied here to llmman’s
PATH-optional llama-server dependency instead.
Prometheus text-format metrics for llmman serve — the data behind
its GET /metrics route (see cmd::serve::handle_metrics), which
only exists when LLMMAN_METRICS=1.
Reusable “resolve a local OCI-store reference to servable model files”
logic — the CNCF ModelPack (https://github.com/modelpack/model-spec)
equivalent of huggingface_hub.snapshot_download.
A JSON object that keeps its members in order, which serde_json’s own
map does not. /v1/systemone needs it end to end: questions are read in
the order written and answers come back in it.
The models.dev provider catalog, fetched at runtime rather than baked
into this binary, so a provider models.dev gains tomorrow works
without an llmman release.
Model sources that are neither an OCI registry nor the HuggingFace
Hub. Ported from the Go shim’s uri_sources.go, whose deletion also
removed the last caller of hf.go.
Speech recognition on the ggml library of the pinned llama.cpp release.
model reads the Canary GGUF description, checks its tensor contract
and uploads the weights onto a mediagen::backend::Backend. wav
and mel turn PCM16 WAV input into the log-mel features the encoder
reads. encoder builds the FastConformer encoder as a ggml graph on
that backend, with the host-side arithmetic it consumes: subsampled
frame count, relative-position table and the folded BatchNorm.
prompt builds the canary2 decoder prompt from the hyperparameters
and the vocabulary; decoder runs the Transformer decoder over the
encoder output with a KV cache; Vocab::detokenize turns decoder
output back into text. server is the worker process’s HTTP side:
it parses and bounds a transcription request, runs that sequence
one request at a time and answers in the OpenAI shapes.
Native <think>...</think> extraction — a direct port of ollama’s
thinking/parser.go, used as a fallback when the inference backend
doesn’t already separate reasoning from content itself (see
cmd::serve::stream’s own oai_chunk_to_content, which prefers a backend’s
structured reasoning_content/thinking delta field whenever one is
present and only ever falls back to this parser scanning raw content
text for literal tag characters).
The web UI’s files, gzipped into the binary by build.rs as the
ASSETS table: (path, content-type, etag, bytes), keyed by the path
relative to webui/. Served by cmd::serve::webui.
Real Xet-protocol downloads for HuggingFace files, via hf-xet
(huggingface/xet-core, Apache-2.0). huggingface_hub’s own client
refuses to fetch any file over 50GB through a plain HTTP GET at all
(file_download.py: “Install hf_xet … for xet-powered
downloads.”), requiring this protocol instead.
Prints a [llmman] [debug] ... line to stderr, only when
debug_enabled. A macro (not a plain function) so the format!
args aren’t even evaluated when debug logging is off.
~/.local/share/llmman (%LOCALAPPDATA%\llmman on Windows): the
parent of the default store and of everything else llmman keeps on
disk for itself — the llama.cpp release (llama_release), the
uv binary and mlx-lm environment (mlx_release), each in the
shape managed describes. Not affected by LLMMAN_MODELS, which
relocates only the store.
Whether verbose diagnostic logging is enabled, from LLMMAN_DEBUG
(mirrors Ollama’s OLLAMA_DEBUG). llmman has only one verbosity
tier, so any truthy value or non-zero integer enables it (Ollama’s
OLLAMA_DEBUG=2 TRACE spelling also works, just maps to the same
tier).
Path to the extracted-model cache — a sibling of the store
(<store>/../cache), matching what llmman serve (the long-running
server that extracts most models) uses. The single canonical answer for
“where is the cache”, so extraction and the GC sweep agree on it rather
than one command writing to <store>/cache and another cleaning
<store>/../cache.
Path to the local OCI store, from LLMMAN_MODELS (mirrors Ollama’s
OLLAMA_MODELS) or else ~/.local/share/llmman/store
(%LOCALAPPDATA%\llmman\store on Windows).