modelc

modelc is a Rust-based CLI that packages model files into single, self-contained artifacts optimized for size, footprint, and performance. It supports multiple model formats and provides an inference experience similar to Ollama, vLLM, and SGLang — run models locally with minimal setup.

Why modelc

• Single-file packaging. One file contains all model data and can be loaded directly by the CLI. No external weight trees, no path coordination, no "wrong checkpoint" drift.
• Size and performance optimized. The packaged format is designed for minimal footprint and fast loading. Parsing and tensor layout happen at package time, so runtime overhead is low.
• Cross-platform. Runs on macOS, Windows, and Linux from the same toolchain.
• Apple Silicon acceleration. Native support for Apple Silicon M-series chips via Metal acceleration.
• Multiple format support. Compatible with Safetensors, GGUF, ONNX, and PyTorch checkpoints.
• Simple operations. modelc run, modelc inspect, modelc compile — a minimal command set for packaging, inspection, and serving.

When weight files remain the better fit: rapid A/B swaps without repackaging, very large checkpoints where embedding blows up artifacts, multitenant “one server, many paths,” or ecosystems that assume on-disk formats (mmap, GGUF loaders, ONNX Runtime with external weights).

See SPEC.md for scope and limits.

Prerequisites

• Rust toolchain with cargo on your PATH (the compiler runs cargo build on a generated project).

Build

Produce the modelc binary:

cargo build --release

Binary path: ./target/release/modelc (or target/debug/modelc without --release). To put modelc on your PATH:

cargo install --path .

Test

cargo test

Examples

Runnable programs live under examples/; see examples/README.md for a table and quick flows. Typical invocations:

cargo run --example create_simple_model -- ./demo_mlp.safetensors
cargo run --example parse_weights -- ./demo_mlp.safetensors safetensors
cargo run --example runtime_inference

Usage

Examples use the modelc command. Put it on your PATH with cargo install --path ., or after cargo build --release call the binary by path (see Build).

Core workflow

Pack models into .modelc single-file artifacts:

modelc pack path/to/model.safetensors -o my-model.modelc
modelc pack path/to/model.gguf --compress -o my-model.modelc
modelc pack path/to/model.safetensors --quantize int8 --prune 0.001 -o my-model.modelc

Run inference from local artifacts:

modelc run my-model.modelc
modelc run my-model.modelc --port 8080 --profile
modelc run my-model.modelc --max-context 2048 --grammar "^\d+$"
modelc run my-model.modelc --temperature 0.7 --max-tokens 512
modelc run my-model.modelc --api-key sk-xxx --rate-limit 120
modelc run my-model.modelc --max-concurrent 8

List locally stored models:

modelc list

Remove a model from the local store:

modelc rm my-model
modelc rm my-model --all          # also delete versioned copies
modelc rm my-model --force        # delete even if versioned copies exist

Copy a model in the local store:

modelc cp my-model my-model-copy

Pull models from remote sources:

modelc pull username/model-name
modelc pull username/model-name --version 2

Chat / completion inference:

# HTTP API: POST /chat with messages
# HTTP API: POST /complete with prompt
# HTTP API: POST /chat/stream for SSE streaming

Legacy compile workflow

Inspect weights (tensor names, shapes, dtypes, sizes):

modelc inspect path/to/model.safetensors
modelc inspect path/to/file -f gguf
modelc inspect path/to/model.safetensors --quant-sizes   # preview fp32/fp16/int8/int4/q4_0 sizes

Compile to a standalone binary (default output: <stem>_serve next to the input):

modelc compile path/to/model.safetensors -o ./my-model-serve

From the built artifact in this repository (no install):

./target/release/modelc inspect path/to/model.safetensors
./target/release/modelc compile path/to/model.safetensors -o ./my-model-serve

modelc --help and modelc --version show subcommands and semver plus a short git revision (from build.rs when .git is present).

Pack flags

• --compress — use zstd compression to reduce artifact size (produces version 2 format).
• --arch — optional hint (llama, gpt2, mlp, …) stored in the model.
• --format / -f — weight format when extensions or magic-byte sniffing are ambiguous.
• -o — output path for the .modelc artifact.

Run flags

• --port — HTTP server port (default 8080).
• --bind — HTTP server bind address (default 0.0.0.0).
• --api-key <key> — Require Authorization: Bearer <key> on all endpoints except /health and /info.
• --rate-limit <N> — Max requests per minute per client IP (default: unlimited).
• --max-concurrent <N> — Max concurrent inference requests. Excess requests receive 503 Service Unavailable. Exempts /health, /info, and /metrics.
• --max-context <N> — Hard context limit; triggers sliding-window KV eviction when exceeded.
• --anchor-tokens <N> — Preserve the first N tokens during context shifting (StreamingLLM-style).
• --temperature <T> — Sampling temperature (default 1.0).
• --max-tokens <N> — Max tokens to generate (default 256).
• --top-p <P> — Nucleus sampling threshold (default 0.0, disabled).
• --min-p <P> — Min-p sampling threshold (default 0.0, disabled).
• --repetition-penalty <F> — Multiplicative repetition penalty (default 1.0, disabled).
• --presence-penalty <F> — OpenAI-style presence penalty (default 0.0, disabled).
• --frequency-penalty <F> — OpenAI-style frequency penalty (default 0.0, disabled).
• --grammar <pattern> — Regex grammar constraint applied during sampling.
• --profile — Print per-step inference timing.

Compile flags (legacy)

The generated model-serve binary binds to --bind (IP, default 0.0.0.0) plus --port (default 8080), unless --listen ADDR:PORT is set, which wins and is embedded verbatim (IPv6 literals such as [::1]:8080 are supported).

Other compile flags:

• --arch — optional hint (llama, gpt2, …) stored in the model and surfaced in /info.
• --format / -f — weight format when extensions or magic-byte sniffing are ambiguous.
• --target — passed through to cargo build --target.
• --debug — builds the generated crate with Cargo’s debug profile instead of --release (release is the default).

Supported input formats (-f when needed):

Ambiguous files (e.g. extensionless or generic .bin): the CLI may sniff GGUF / zip (PyTorch-ish) / small Safetensors blobs (see SPEC.md).

HTTP API (run + model-serve)

Inference backends (priority order):

1. ONNX execution plan — if the model metadata contains onnx.execution_plan, ops are executed via the runtime tensor engine (MatMul, Gemm, Add, Mul, Div, Sub, Relu, Softmax, LayerNorm, Transpose, Reshape, Sigmoid, Tanh, Identity, Cast).
2. Transformer forward — when architecture == "gpt2" or "llama", runs the full FP32 transformer forward (layer norm / RMS norm, GeLU / SwiGLU, RoPE, single-token causal attention, output projection). Inputs are resized to the model hidden size; numerics match the compile-emitted server.
3. MLP GEMV — when architecture == "mlp", emits a stacked GEMV + bias (+ ReLU between hidden layers) using layerN.weight/layerN.bias or a single weight/bias.
4. Echo fallback — returns input unchanged when no execution plan matches.

All JSON responses are application/json; SSE uses text/event-stream.

crates.io checklist

Before the first (modelc) publish:

1. Bump Cargo.toml version, tag vX.Y.Z, run cargo publish --dry-run.
2. Snapshot modelc --help / subcommand help after any CLI churn (crate README can embed the summary).
3. After the first publish, add a crates.io version badge (shields.io versioning; crate URL https://crates.io/crates/modelc once live).
4. Maintain a concise changelog (CHANGELOG.md optional) noting parser/format support changes.

Format references (parsers / exports)

• Safetensors — huggingface/safetensors
• GGUF — GGML GGUF notes
• ONNX — onnx.ai
• PyTorch checkpoints — accepts mislabeled standalone Safetensors bytes and Torch ZIP containers that nest *.safetensors payloads; pickled-only checkpoints should be exported to Safetensors, ONNX, or GGUF externally.

Features

• ONNX graph execution — parses ONNX graph nodes into an execution plan (MatMul, Gemm, Add, Mul, Div, Sub, Relu, Softmax, LayerNorm, Transpose, Reshape, Sigmoid, Tanh, Identity, Cast) and runs inference via the tensor runtime.
• LoRA adapter support — load and apply LoRA adapters on top of a base model at runtime (src/lora.rs).
• INT4 quantization + weight pruning — pack-time --quantize int4 and --prune <threshold> for extreme size reduction.
• Docker/OCI image generation — modelc containerize <artifact> emits a minimal Dockerfile + entrypoint.
• Model versioning — store multiple versions and modelc switch <name> <version>.
• Shell completions — generate bash/zsh/fish completions for all subcommands.
• Per-op profiling — --profile flag on run prints timing per inference step.
• KV cache — KvCache stores per-layer Key/Value vectors for autoregressive generation. forward_gpt2_cached / forward_llama_cached append current K/V and compute attention over all cached positions, eliminating redundant recomputation.
• Autoregressive text generation — src/generate.rs provides generate() with greedy, temperature, and top-p (nucleus) sampling, KV cache reuse, and token embedding lookup. Wires into the HTTP server so /chat, /complete, and /v1/chat/completions return real generated text for GPT-2 / LLaMA models. Accepts per-request max_tokens, temperature, and top_p overrides on all text generation endpoints.
• Byte-level BPE tokenizer — src/tokenizer.rs provides encode/decode with greedy merge algorithm. Foundation for real text-in/text-out transformer inference.
• GGUF tokenizer metadata extraction — reads vocab, merges, and BOS/EOS token IDs directly from GGUF KV metadata (extract_tokenizer_metadata). Constructs a BpeTokenizer from in-file tokenizer data without external files.
• Chat template rendering — src/chat_template.rs reads Jinja2 templates from GGUF metadata (tokenizer.chat_template) and formats messages before tokenization using minijinja. Falls back to simple concatenation when no template is present.
• GGUF quantization inference — Q4_0, Q5_0, Q8_0, Q4_K, Q6_K block-quantized tensors are preserved in IR and dequantized on-the-fly by Runtime::from_raw (src/runtime/serve.rs). Reduces .modelc artifact size ~8x for Q4_0 models while keeping the transformer forward pass functional. Codegen path (compile) calls dequantize_in_place before emitting the server so generated binaries still receive FP32 weights.
• Structured output / JSON mode — POST /v1/chat/completions accepts response_format: { type: "json_object" } to constrain output to valid JSON. Injects a system prompt requesting JSON-only responses and post-processes generated text with extract_json_object to extract the first well-formed JSON object or array. Falls back to raw text if no valid JSON is found.
• Function calling / tool use — POST /v1/chat/completions accepts an OpenAI-compatible tools array. When tools are present, a system prompt describing available tools is injected into the conversation. Generated output is parsed for a tool_calls JSON array; if found, the response returns tool_calls with finish_reason: "tool_calls". Otherwise falls back to regular text content. Supports both inline JSON arguments and pre-serialized argument strings.
• Continuous batching (MLP) — POST /infer with multiple inputs and MLP architecture routes to run_mlp_forward_batched (src/serve/infer.rs), which computes the entire batch in a single pass via batched_gemv_bias. Eliminates redundant weight traversal and improves CPU cache locality compared to serial per-item inference. Transformer generation batching is future work.
• Speculative decoding — generate() supports config.gamma > 0 to enable speculative decoding with an n-gram draft model (src/generate.rs). The draft model proposes gamma candidate tokens by looking up trigram continuations from the prefix; the target model verifies each candidate in a loop using the KV cache. Accepted tokens skip the sampling step. Now supports prefix caching (restores cached K/V before priming), context shifting (max_context), and grammar constraints (sample_constrained). Disabled by default (gamma: 0). Establishes infrastructure for future faster draft models (e.g., smaller transformer or prompt lookup decoding).
• Streaming token generation — /chat/stream and /v1/chat/completions with stream: true generate token IDs via generate_token_ids(), then decode and emit each token incrementally through SSE. After each new token, the cumulative sequence is decoded and only the text delta is sent. The OpenAI-compatible stream emits chat.completion.chunk objects with delta.role on the first chunk and delta.content per token, ending with finish_reason: "stop" and a [DONE] sentinel. Non-transformer models fall back to a single chunk.
• Batched embeddings — POST /embeddings accepts inputs: ["...", "..."] and returns embeddings: [{"embedding": [...], "index": 0}, ...]. Single-input input: "..." remains backward compatible and returns embedding: [...].
• Token-level logprobs — POST /v1/chat/completions accepts OpenAI-compatible logprobs: true and top_logprobs: N (0–20). generate_with_logprobs() (src/generate.rs) records ln(P(token)) from the raw pre-temperature softmax plus up to N top alternatives per position, so logprobs are deterministic regardless of sampling temperature. The response carries choices[].logprobs.content with per-token token, logprob, raw UTF-8 bytes, and top_logprobs entries. The field is null when not requested (matching OpenAI); non-transformer models return an empty content array.
• Quantized KV cache — KvLayer::Int8 (src/runtime/transformer.rs) stores per-token K/V vectors as INT8 with per-token scales, giving ~4× memory reduction vs FP32. KvCache::new_quantized(n_layers, hidden, use_int8) selects INT8 or FP32 at creation; GenerationConfig.use_int8_kv wires the choice through generate_token_ids, generate_with_logprobs, and speculative_generate. Dequantization is transparent via k_all/v_all, so attention and generation logic is unchanged.
• Prefix caching — PrefixCache (src/prefix_cache.rs) stores KvCache snapshots keyed by token sequence. Longest-prefix matching lets later requests whose prompt starts with a cached sequence restore the saved K/V and only process the divergent suffix. Wired into generate_core, generate_token_ids_with_cache, and generate_with_logprobs_with_cache. The serve layer locks a per-state cache and passes it through. LRU eviction bounds memory to 32 entries.
• LoRA loading at runtime via HTTP — POST /lora/load { path: "...", alpha: 1.0 } clones the base model tensors, applies a Safetensors LoRA adapter (src/lora.rs), and atomically swaps the runtime via RwLock<Runtime>. POST /lora/unload restores the base model from the stored base_tensors without restarting the server.
• Grammar-based constrained decoding — GenerationConfig.constraint holds an optional Arc<dyn Constraint>. RegexConstraint (src/constraint.rs) wraps a regex pattern and masks invalid tokens to -inf before sampling. The sample_constrained helper in generate.rs decodes the partial output, builds a per-token validity mask, and applies it. HTTP endpoints (/chat, /complete, /v1/chat/completions) accept an optional grammar string (regex pattern) per request. The heuristic is permissive (tries common suffixes to avoid false negatives) rather than exact, which is acceptable for a minimal implementation.
• Prometheus metrics — GET /metrics renders request counts, inference latency histograms (buckets: 0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, +Inf), active request gauge, and tokens-generated counter in Prometheus text format. Handlers use ActiveRequestGuard and InferenceTimer RAII structs so instrumentation is automatic and zero-overhead when not scraped.
• JSON Schema constrained generation — POST /chat, /complete, and /v1/chat/completions accept an optional json_schema object per request. Generated text is parsed as JSON and validated against the schema using the jsonschema crate. If invalid, generation is retried up to 3 times with increasing temperature to encourage diversity. More robust than the post-hoc json_object mode (response_format: { type: "json_object" }), which still works for backward compatibility.
• Context shifting (sliding window KV cache) — GenerationConfig.max_context sets a hard context limit. When total tokens exceed it during generation, generate_core shifts both the KvCache (KvLayer::shift removes oldest K/V vectors from every layer) and token_ids left, keeping roughly the newest 75%. prompt_len is adjusted so generation continues seamlessly. Enables effectively infinite context length at the cost of losing distant history.
• Mixed-precision KV cache — KvLayer::Mixed stores recent tokens in FP32 and automatically moves older tokens to INT8 cold storage when the hot window (default 64 tokens) overflows. Preserves accuracy for the actively attended context while reducing memory footprint vs pure FP32. Compatible with prefix caching, context shifting, and speculative decoding.
• Anchor token preservation (StreamingLLM-style) — GenerationConfig.anchor_tokens (default 0, set via --anchor-tokens N) preserves the first N initial tokens during context shifting. The first few tokens act as "attention sinks"; evicting them degrades generation quality. KvLayer::shift_anchored removes tokens from the middle (after anchors) rather than from the beginning. Works with all KV cache variants (FP32, INT8, Mixed).
• Concurrent transformer generation — The prefix cache was a Mutex that serialized all transformer requests. Refactored to RwLock with brief lock hold times: read-lock for lookup, release during generation, write-lock for insertion. Multiple transformer requests now run in parallel, limited only by CPU cores. Covered by run_server_handles_concurrent_transformer_requests test.
• Attention allocation optimization — attention_kv previously allocated ~3×n_heads buffers per attention call (e.g., ~432 allocations per token for a 12-layer, 12-head model). Rewrote to use a single pre-allocated scratch buffer reused across heads, plus softmax_inplace with zero allocations. Eliminates the dominant allocation hotspot in the transformer forward pass.
• OpenAI /v1/completions endpoint — Legacy (non-chat) completions API: POST /v1/completions { model, prompt, max_tokens, temperature, top_p, stop, stream }. Returns text_completion objects with choices[].text. Supports streaming SSE, logprobs, top_logprobs, grammar, and json_schema.
• EAGLE3 / neural speculative decoding — src/draft.rs introduces a DraftModel trait and an MlpDraftModel: a tiny 2-layer MLP (embedding → FC1 + ReLU → FC2 → logits) that is orders of magnitude faster than the full transformer. Falls back to PromptLookupDraftModel (weight-free, searches the token context for repeated substrings) when no neural draft tensors are present. Wired into all serve endpoints automatically.
• API key authentication + rate limiting — modelc run accepts --api-key <key> (requires Authorization: Bearer <key> on all endpoints except /health and /info) and --rate-limit <N> (max requests per minute per client IP). Returns standard HTTP status codes (401 Unauthorized, 429 Too Many Requests).
• Stop sequences — GenerationConfig.stop holds a list of strings that halt generation when any appear in the decoded output. Checked after each token; output is truncated to just before the matched sequence. Wired through all text generation endpoints.
• Repetition / presence / frequency penalties — three sampling penalties discourage token reuse. repetition_penalty (multiplicative, transformers-style) and OpenAI-standard presence_penalty (once per seen token) / frequency_penalty (scales with count) adjust logits before sampling. Wired through all text-generation endpoints (/chat, /complete, /v1/chat/completions, /v1/completions) as optional request fields and the CLI (--repetition-penalty, --presence-penalty, --frequency-penalty). They share a fast path that skips the per-token allocation when all are at their defaults.
• Logit bias — OpenAI-standard logit_bias parameter: a JSON object mapping token ID to additive bias (-100 to 100). Positive values boost a token's likelihood; negative values suppress it. Applied after penalties but before grammar constraints. Wired through all text-generation endpoints as an optional logit_bias request field. Empty map = fast path (no allocation).
• echo and n for /v1/completions — OpenAI-standard parameters: echo: true prepends the prompt to the completion in the response (streaming emits the prompt as the first chunk); n: N generates N independent completions (clamped 1–20) with incrementing index in choices. n is also supported on /v1/chat/completions.
• /props server properties endpoint — GET /props (llama.cpp-compatible) exposes model name, architecture, total params/bytes, chat template, and default generation parameters so clients can auto-configure without trial-and-error. Exempt from auth and backpressure.
• /detokenize endpoint — POST /detokenize (llama.cpp-compatible) decodes token IDs back to text, complementing /tokenize for round-trip token management.
• suffix for /v1/completions — OpenAI-standard suffix parameter appends text after the generated completion. With echo: true, output is prompt + completion + suffix.
• stream_options.include_usage — OpenAI-standard streaming option for /v1/chat/completions and /v1/completions. When stream_options: { include_usage: true } is set, the SSE stream emits a final chunk with empty choices and a usage object before [DONE].
• GET /v1/models/:id — OpenAI-standard model retrieval endpoint. Returns a single model object by ID, or 404 model_not_found for unknown IDs.
• best_of for /v1/completions — OpenAI-standard best_of generates N completions server-side (clamped 1–20), picks the best by token count, and returns the top n.
• user field — OpenAI-standard user parameter accepted on both /v1/chat/completions and /v1/completions for abuse tracking. Accepted but ignored.
• response_format: { type: "json_schema" } — OpenAI-standard Structured Outputs for /v1/chat/completions. Injects the schema as a system prompt and validates/retries generated output against it.
• tool_choice for /v1/chat/completions — OpenAI-standard tool choice control: "auto" (default), "none" (no tools), "required" (must call a tool), or {"type": "function", "function": {"name": "..."}} (must call named tool).
• name field on messages — OpenAI-standard name field on chat message objects, accepted and serialized in responses.
• max_completion_tokens — OpenAI's newer parameter name for max_tokens (o1+ models). Accepted on both endpoints; takes precedence over max_tokens when both are provided.
• Batch prompt for /v1/completions — OpenAI-standard batch mode. When prompt is an array of strings, generates completions for each prompt with incrementing index.
• Min-p sampling — min_p keeps only tokens whose probability is at least min_p fraction of the max probability, then renormalizes. The threshold scales with the model's confidence per step, making it simpler and often more effective than top-p (nucleus). Can combine with top_p (min-p runs after). Wired through all text-generation endpoints as min_p and the CLI (--min-p).
• Tokenize endpoint — POST /tokenize returns token IDs and count for a prompt (single input or batch inputs), using the same byte-level BPE tokenizer as /chat. Standard in Ollama and llama.cpp server; useful for prompt budgeting and management.
• System info endpoint — GET /v1/system exposes best-effort hardware info (CPU cores, OS, arch, pointer width, Metal availability, total memory) for orchestration and debugging. No added dependencies (stdlib + /proc/meminfo/sysctl).
• Quantization size preview — modelc inspect --quant-sizes previews the artifact size under each format (fp32/fp16/int8/int4/q4_0) and the savings vs the current size, without actually quantizing. Computed from element counts, so it's accurate regardless of current dtype.
• Request cancellation on disconnect — when an SSE streaming client (/chat/stream, /v1/chat/completions, /v1/completions with stream: true) disconnects, the server trips a cooperative cancel flag checked once per token, aborting the generation loop instead of running to completion. Saves CPU when users stop a response mid-stream.

Repository layout

• src/ — CLI, parsers, Model IR, codegen, runtime helpers, ONNX execution engine.
  • src/parsers/ — format-specific parsers, modularized by format (gguf/, onnx/ subdirectories).
  • src/onnx_exec/ — ONNX graph execution plan builder and executor (mod.rs, helpers.rs).
  • src/codegen/native/ — native code generator, modularized (mod.rs, forward.rs, helpers.rs).
  • src/serve/ — HTTP inference server, modularized (mod.rs, handlers.rs, infer.rs).
  • src/tokenizer.rs — byte-level BPE tokenizer (encode/decode with greedy merge algorithm).
  • src/chat_template.rs — Jinja2 chat template rendering for message formatting before tokenization.
• examples/ — runnable cargo --example programs (examples/README.md).
• tests/ — integration tests.

See SPEC.md, ARCHITECTURE.md, and TODO.md.

License

Licensed under the Apache License, Version 2.0 (LICENSE).