# SPEC — modelc
## Purpose
**modelc** is a Rust-based command-line tool and library that:
1. **Loads** neural network **weights** from supported on-disk formats (Safetensors, GGUF, ONNX, PyTorch).
2. **Normalizes** them into an internal **`Model`** IR (named tensors with shape, dtype, and raw bytes).
3. **Packages** them into a **single optimized artifact** that contains all model data and can be loaded by the CLI for inference.
4. **Runs** inference locally with an HTTP API, optimized for size, footprint, and performance.
The primary user outcome: a **single file** per model that can be inspected, moved, and run without Python or the original framework at runtime — similar to the experience of Ollama, vLLM, or SGLang.
## Non-goals (current scope)
- Full graph capture and arbitrary framework-accurate inference for every ONNX or PyTorch export (parsers focus on **weights** into the IR; ONNX execution covers a core subset of ops; generated servers emit a real **FP32 MLP** forward when `--arch mlp` matches `layerN.(weight,bias)` or `weight`/`bias`, otherwise inference falls back to ONNX plan or echoes input).
- Training or fine-tuning.
- Replacing general-purpose inference servers (e.g. full ONNX Runtime) for models outside the supported op set.
## User-facing commands
### `pack`
**Input:** path to weights file; optional `-f`, `-o`, `--arch`, `--compress`, `--quantize fp16|int8|int4`, `--prune <threshold>`.
**Output:** `.modelc` single-file artifact containing JSON header + tensor blob (optionally compressed / quantized / pruned).
**Side effects:** writes artifact file to specified path.
**Errors:** parse failure; invalid output path; compression failure.
### `run`
**Input:** path to `.modelc` artifact (or uses default store); optional `--port`, `--bind`, `--profile`.
**Output:** HTTP server listening for inference requests.
**Side effects:** loads artifact into memory, starts HTTP server.
**Errors:** missing artifact; invalid artifact format; server bind failure.
### `list`
**Input:** none (uses default model store).
**Output:** stdout list of locally stored `.modelc` artifacts with metadata.
**Errors:** store access failure.
### `pull`
**Input:** model identifier (e.g., `username/model-name`); optional `--version`.
**Output:** downloaded `.modelc` artifact stored locally.
**Errors:** network failure; invalid model identifier.
### `search`
**Input:** query string.
**Output:** filtered list of models matching name or architecture.
### `bench`
**Input:** path to `.modelc` artifact.
**Output:** warm/cold latency and throughput measurements.
### `containerize`
**Input:** path to `.modelc` artifact; optional `--base-image`.
**Output:** Dockerfile + entrypoint.sh in target directory.
### `switch`
**Input:** model name, version number.
**Output:** activates specified version in store.
### `inspect`
**Input:** path to a weights file; optional `-f` / `--format`.
**Output:** stdout summary — model name, architecture string, format name, parameter count, total size, sorted tensor list, optional metadata (`__metadata__` from Safetensors is merged when present).
**Errors:** missing or undetectable format; parse failure.
### `compile` (legacy)
**Input:** path to weights; optional `-f`, `-o`, `--arch`, `--bind`, `--port`, **`--listen ADDR:PORT`** (overrides bind+port), `--target`, **`--debug`** (use debug profile on the emitted crate instead of `--release`).
**Output:** path to the built executable (copied from the generated crate’s `target/{...}/model-serve`).
**Side effects:** temp directory, generated `modelc_build` project, `cargo build`, copy binary, chmod on Unix.
**Errors:** parse failure; invalid `--listen` / `--bind`; `cargo` failure.
**Version line:** each `compile` invocation prints `modelc <semver> (git <sha>)` to stderr (semver from Cargo, SHA from `build.rs`, or `unknown` without git).
## Supported weight formats (declared)
| Safetensors | Implemented (`safetensors` crate). |
| GGUF | Implemented for **F32/F16/BF16/F64/I8/I16/I32/I64** and contiguous integer blobs; **Q4_0, Q5_0, Q8_0, Q4_K, Q6_K** blocks are **dequantized to F32** in IR. Unsupported quant types still error with a named type hint. |
| ONNX | Implemented for **initializer** tensors: inlined `raw_data` / typed fields, **external** payloads (`external_data` with `location`/`offset`/`length`), and **segmented** initializers. Also parses the graph into an `ExecutionPlan` stored as `onnx.execution_plan` metadata. |
| PyTorch | Implemented for **Safetensors-in-ZIP** (and standalone Safetensors mislabeled `.pt`/`.pth`); pickle-only checkpoints need export outside `modelc`. |
## Format detection
1. **Extension** (`.safetensors`, `.gguf`, `.onnx`, `.pt`, `.pth`, heuristics on `.bin`).
2. **Sniffing** when needed: `GGUF` prefix, zip `PK\x03\x04` (common for torch archives), or full-file Safetensors parse for files **≤ 64 MiB** when the extension is ambiguous or missing.
3. Otherwise detection fails and the user must pass `-f`.
## Internal model contract
- **`Model`**: `name`, `architecture`, `tensors: HashMap<String, TensorData>`, `metadata: HashMap<String, String>`.
- **`TensorData`**: `shape`, `dtype` (`DataType` enum), `data: Vec<u8>`.
Serialization via `serde` for tests and tooling.
## Artifact formats
### `.modelc` (primary)
Single-file binary format with:
- **JSON header** — model metadata (name, architecture, version, compression flag)
- **Tensor blob** — concatenated tensor data in sorted order
- **Compression** — optional zstd compression (version 2 format)
Created by `pack`, consumed by `run`.
### `model-serve` binary (legacy)
- Rust edition **2021** in the emitted crate; **axum**, **tokio**, **serde**, **serde_json**.
- Static bytes: `include_bytes!("../embedded_weights.bin")` built from the IR (sorted tensor names, streamed concatenation straight to disk) so `byte_offset` / `byte_len` match the blob **without retaining a second full in-memory blob copy** during codegen.
- **Listen address** parsed at runtime from a string literal produced from `modelc compile` (`--listen` or `--bind` + `--port`).
### HTTP API (both `run` and `model-serve`)
- **`GET /info`** — JSON object: `name`, `architecture`, `total_params`, `total_bytes`, `tensors` (array of tensor name strings).
- **`POST /infer`** — request JSON `{ "input": number[] }` (`f32`) or `{ "inputs": [[...], [...]] }` for batch. Response `{ "output": number[] }` or `{ "outputs": [[...], [...]] }`.
- **`POST /chat`** — request JSON `{ "messages": [{"role": "...", "content": "..."}] }`, response `{ "message": {"role": "assistant", "content": "..."} }`.
- **`POST /chat/stream`** — SSE stream of `{ "delta": "...", "done": bool }` chunks.
- **`POST /complete`** — request JSON `{ "prompt": "..." }`, response `{ "completion": "..." }`.
**Inference pipeline** (in order of preference):
1. **ONNX execution plan** — if `onnx.execution_plan` metadata exists, ops are executed via the tensor runtime.
2. **MLP GEMV** — when `architecture == "mlp"`, runs stacked GEMV + bias (+ ReLU between hidden layers) using `layerN.weight`/`layerN.bias` or a single `weight`/`bias`.
3. **Echo fallback** — returns input unchanged.
## Library runtime
`modelc::runtime::ops` exposes tensor helpers (`matmul`, `linear`, `softmax`, …). `Runtime::from_raw` loads tensors from `TensorData` into `f32` buffers for **F32, F16, BF16, I64, I32, I16, I8, U8, Bool** (integer/bool values are cast to `f32` for the internal representation).
## Platform support
- **macOS** — primary development and runtime target; Apple Silicon M-series acceleration via Metal.
- **Windows** — runtime target; CLI and generated server build and run.
- **Linux** — runtime target; CLI and generated server build and run.
## Acceleration
- **Apple Silicon (M-series):** Full Metal GPU kernels for relu, add, mul_scalar, softmax, layer_norm, matmul (`src/metal.rs`, `src/compute/shaders.metal`). GPU memory pressure handling via pre-allocation checks.
- **CPU:** AVX (x86_64) and NEON (aarch64) SIMD paths with runtime feature detection; `rayon` parallelization for large matmuls.
## Compatibility
- **Host:** `cargo` required for `compile`.
- **Targets:** optional `--target` triple for the generated build.
## Success criteria
- `cargo test` and CI (`fmt`, `clippy -D warnings`, `test`) pass.
- `inspect` / `pack` / `run` succeed on supported fixtures and examples where parsers are implemented.
- `.modelc` artifact is a **single file** containing all model data.
- `list` correctly enumerates locally stored artifacts.
- Artifact size and load time are optimized relative to raw weight directories.