# TODO — Plan
## Implemented
- [x] **Parsers** — GGUF (F32/F16/BF16/I8/I16/I32/I64/F64, Q4_0/Q5_0/Q8_0/Q4_K/Q6_K dequant), ONNX (inline/external/segmented initializers), PyTorch (Safetensors-in-ZIP), Safetensors (`src/parsers/`).
- [x] **ONNX graph execution** — parses ops (MatMul, Gemm, Add, Mul, Div, Sub, Relu, Softmax, LayerNorm, Transpose, Reshape, Sigmoid, Tanh, Identity, Cast) into a JSON execution plan stored in metadata (`src/onnx_exec/`).
- [x] **MLP codegen forward** — FP32 GEMV chain when `architecture == "mlp"` (`src/codegen/native/`).
- [x] **GPT2 / LLaMA codegen skeletons** — simplified transformer forward pass stubs (`src/codegen/native/forward.rs`).
- [x] **Auto architecture inference** — detects `gpt2`, `llama`, `bert`, `mlp` from tensor naming patterns (`src/model.rs`).
- [x] **Compile peak RAM** — Embedding writes stream via `BufWriter` (no auxiliary Vec blob).
- [x] **crates.io checklist** — README subsection with publish steps.
- [x] **Single-file artifact format** — `.modelc` binary with JSON header + raw tensor blob (`src/pack.rs`).
- [x] **Cross-platform runtime** — Platform-specific store paths via `dirs` crate; `#[cfg]` Unix/Windows guards.
- [x] **Apple Silicon Metal acceleration** — Full kernels for relu, add, mul_scalar, softmax, layer_norm (`src/metal.rs`, `src/compute/shaders.metal`).
- [x] **Ollama-like UX** — `pack`, `run`, `pull`, `list` with local model store (`src/store.rs`).
- [x] **Size optimization** — `--compress` flag on `pack` uses zstd (version 2 format).
- [x] **Quantization at pack time** — `--quantize fp16/int8/int4` converts F32 tensors; INT4/INT8 auto-dequantized on `run`.
- [x] **Weight pruning** — `--prune <threshold>` zeroes small weights.
- [x] **CPU SIMD matmul** — AVX (x86_64) and NEON (aarch64) paths with runtime feature detection.
- [x] **Multi-threaded CPU ops** — `rayon` parallelization for large matmuls.
- [x] **GPU memory pressure handling** — pre-allocation checks before Metal buffer creation.
- [x] **Store metadata enrichment** — `modelc list` reads artifact headers to show architecture, params, and compression status.
- [x] **Remote model pull** — `modelc pull <url>` supports HTTP/HTTPS downloads with versioning.
- [x] **Benchmark command** — `modelc bench <artifact>` measures warm/cold latency and throughput.
- [x] **Artifact verification** — `modelc verify <artifact>` checks header integrity.
- [x] **Export to Safetensors** — `modelc export <artifact> -o out.safetensors`.
- [x] **Model search** — `modelc search <query>` filters store by architecture, size, or name.
- [x] **Config file support** — `~/.modelc/config.toml` for default bind, store path, compression.
- [x] **Shell completions** — bash/zsh/fish via `modelc completions <shell>`.
- [x] **Per-op profiling** — `--profile` flag on `run` prints per-step timing.
- [x] **Model versioning** — store multiple versions and `modelc switch <name> <version>`.
- [x] **Model card generation** — `modelc inspect --readme` generates Markdown model card.
- [x] **Docker/OCI image generation** — `modelc containerize <artifact>` emits Dockerfile + entrypoint.
- [x] **LoRA adapter support** — load and apply LoRA adapters at runtime (`src/lora.rs`).
- [x] **Chat/completion endpoints** — `/chat`, `/complete`, `/chat/stream` (SSE).
- [x] **Batch inference API** — `/infer` accepts `{"inputs": [...]}`.
- [x] **Code modularization** — split large files into focused submodules (`gguf/`, `onnx/`, `onnx_exec/`, `codegen/native/`, `serve/`).
- [x] **Transformer runtime in `run`** — `modelc run` now executes GPT-2 / LLaMA forward passes directly from the in-memory runtime (`src/runtime/transformer.rs`) instead of echoing input. Structure inspection shared with codegen via `src/arch.rs`; numeric helpers mirror the emitted codegen so `run` and `compile` outputs match.
## Next / Competitive gaps
- [x] **`modelc rm <name>`** — Delete a model from the local store (and its versioned copies). Every model manager (Ollama, Docker) has this; we don't.
- [x] **OpenAI-compatible API** — `/v1/models`, `/v1/chat/completions` (non-streaming). Ollama, llama.cpp server, vLLM, and SGLang all expose this; it's the dominant integration pattern for client libraries.
- [x] **`/embeddings` endpoint** — Produce vector embeddings from the hidden-state pooler via `POST /embeddings`.
- [x] **KV cache in transformer runtime** — `KvCache` struct stores per-layer K/V vectors. `forward_gpt2_cached` and `forward_llama_cached` accept optional cache, append current K/V, and compute attention over all cached positions via `attention_kv`. Eliminates redundant K/V recomputation during autoregressive generation.
- [x] **Built-in tokenization** — Byte-level BPE tokenizer (`src/tokenizer.rs`) with encode/decode and greedy merge algorithm. Foundation for real text-in/text-out transformer inference.
- [x] **GGUF tokenizer metadata extraction** — `extract_tokenizer_metadata` reads vocab, merges, BOS/EOS IDs from GGUF KV metadata. `BpeTokenizer::from_gguf` constructs a working tokenizer from this data.
- [x] **`/health` endpoint** — Standard liveness probe for inference servers and container orchestration.
- [x] **Model deletion guard** — Optional `--force` flag on `rm`; refuse to delete the main artifact if versioned copies exist unless `--all` or `--force` is provided.
## Competitive gaps (research-driven)
- [x] **Autoregressive text generation** — `src/generate.rs` provides `generate()` with greedy and temperature sampling. Wires into `run_text_inference` for transformer models so `/chat`, `/complete`, and `/v1/chat/completions` return real generated text instead of logits JSON.
- [x] **Chat template support** — reads Jinja2 chat templates from GGUF metadata (`tokenizer.chat_template`) and applies them to format messages before tokenization via `minijinja`. Falls back to simple `role: content\n` concatenation when no template is present.
- [x] **GGUF quantization inference** — GGUF parser preserves Q4_0, Q5_0, Q8_0, Q4_K, Q6_K block-quantized bytes in IR instead of expanding to F32 at parse time. `Runtime::from_raw` dequantizes on-the-fly via `dequantize_gguf_tensor`. This reduces `.modelc` artifact size (~8x for Q4_0) while keeping inference functional. Codegen path calls `dequantize_in_place` before generating the server.
- [x] **Structured output / JSON mode** — `response_format: { type: "json_object" }` on `/v1/chat/completions` injects a system prompt instructing JSON-only output, then post-processes generated text with `extract_json_object` to extract the first well-formed JSON object or array. Falls back to raw text if no valid JSON is found.
- [x] **Function calling / tool use** — `POST /v1/chat/completions` accepts an OpenAI-compatible `tools` array. When tools are present, a system prompt describing available tools is injected. Generated output is parsed for a `tool_calls` JSON array; if found, the response returns `tool_calls` with `finish_reason: "tool_calls"`. Otherwise falls back to regular content.
- [x] **Continuous batching (MLP)** — `POST /infer` with multiple `inputs` and MLP architecture now routes to `run_mlp_forward_batched`, which computes all items in a single pass via `batched_gemv_bias`. Eliminates redundant weight traversal and improves cache locality. Transformer generation batching remains future work.
- [x] **Speculative decoding** — `generate()` supports `config.gamma > 0` to enable speculative decoding with an n-gram draft model (`src/generate.rs`). The draft model proposes `gamma` candidate tokens by looking up trigram continuations from the prefix; the target model verifies each candidate in a loop using the KV cache. Accepted tokens skip the sampling step. Establishes infrastructure for future faster draft models (e.g., smaller transformer or prompt lookup decoding).
- [x] **Top-p / nucleus sampling** — `GenerationConfig.top_p` enables nucleus filtering: probabilities are sorted descending, the smallest prefix whose cumulative sum exceeds `top_p` is kept, and the rest are zeroed before renormalization and sampling. Wired through all text generation endpoints (`/chat`, `/complete`, `/v1/chat/completions`) as an optional request parameter.
- [x] **Streaming token generation** — `/chat/stream` now generates token IDs via `generate_token_ids()`, then decodes and emits each token incrementally through SSE. After each new token, the cumulative sequence is decoded and only the text delta (newly added prefix) is sent. `done: true` marks the final chunk. Non-transformer models fall back to a single chunk.
- [x] **Batched embeddings** — `POST /embeddings` now accepts `inputs: ["...", "..."]` and returns `embeddings: [{"embedding": [...], "index": 0}, ...]`. Single-input `input: "..."` remains backward compatible and returns `embedding: [...]`.
- [x] **Token-level logprobs** — `POST /v1/chat/completions` accepts OpenAI-compatible `logprobs: true` and `top_logprobs: N` (0–20). `generate_with_logprobs()` (`src/generate.rs`) drives the autoregressive loop and records `ln(P(token))` from the raw (pre-temperature) softmax plus up to N top alternatives per position. The response carries `choices[].logprobs.content` with per-token `token`, `logprob`, raw UTF-8 `bytes`, and `top_logprobs` entries. Non-transformer models return an empty `content` array; when `logprobs` is omitted the field serializes to `null` (matching OpenAI).
- [x] **Quantized KV cache** — `KvLayer::Int8` (`src/runtime/transformer.rs`) stores per-token K/V as INT8 with per-token scales (4× memory reduction vs FP32). `KvCache::new_quantized(n_layers, hidden, use_int8)` selects INT8 vs FP32 at creation; `GenerationConfig.use_int8_kv` wires the choice through `generate_token_ids`, `generate_with_logprobs`, and `speculative_generate`. Dequantization is transparent via `k_all`/`v_all`. Covered by `kv_layer_int8_roundtrip` / `kv_layer_int8_multiple_tokens` tests.
## Brainstorming (research-driven)
- [x] **Prefix caching** — `PrefixCache` (`src/prefix_cache.rs`) stores `KvCache` snapshots keyed by token sequence. Longest-prefix matching lets later requests whose prompt starts with a cached sequence restore the saved K/V and only process the divergent suffix. Wired into `generate_core`, `generate_token_ids_with_cache`, and `generate_with_logprobs_with_cache`. Serve layer (`run_text_inference_with_config`, `run_text_inference_token_ids`, `run_text_inference_with_logprobs`) locks the per-state cache and passes it through. LRU eviction bounds memory to 32 entries.
- [x] **LoRA loading at runtime via HTTP** — `POST /lora/load { path: "...", alpha: 1.0 }` clones the base model tensors, applies a Safetensors LoRA adapter (`src/lora.rs`), and swaps the runtime atomically via `RwLock<Runtime>`. `POST /lora/unload` restores the base model from the stored `base_tensors`. No server restart required. Covered by `run_server_lora_load_bad_path_returns_error` and `run_server_lora_unload_restores_base` tests.
- [x] **Grammar-based constrained decoding** — `GenerationConfig.constraint` holds an optional `Arc<dyn Constraint>`. `RegexConstraint` (`src/constraint.rs`) wraps a regex and masks invalid tokens to `-inf` before sampling. The `sample_constrained` helper in `generate.rs` decodes the partial output, builds a per-token validity mask, and applies it. HTTP endpoints (`/chat`, `/complete`, `/v1/chat/completions`) accept an optional `grammar` string (regex pattern) per request. The heuristic tries common suffixes to avoid false negatives; it is permissive rather than exact, which is acceptable for a minimal implementation.
- [x] **Prometheus metrics endpoint** — `GET /metrics` exposes `modelc_requests_total`, `modelc_inference_duration_seconds` (histogram with buckets 0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, +Inf), `modelc_active_requests` gauge, and `modelc_tokens_generated_total` counter. Handlers use `ActiveRequestGuard` and `InferenceTimer` RAII structs to track automatically. Rendered in Prometheus text exposition format. Covered by `run_server_metrics_returns_prometheus_text` test.
- [x] **Context shifting (sliding window KV cache)** — `GenerationConfig.max_context` sets a hard context limit. When total tokens exceed it, `generate_core` shifts both the `KvCache` (`KvLayer::shift` removes oldest K/V vectors) and `token_ids` left, keeping roughly the newest 75%. `prompt_len` is adjusted so generation continues seamlessly. Covered by `kv_layer_fp32_shift_discards_oldest`, `kv_layer_int8_shift_discards_oldest`, and `kv_cache_shift_syncs_all_layers` tests.
- [x] **JSON Schema constrained generation** — `POST /chat`, `/complete`, and `/v1/chat/completions` accept an optional `json_schema` object per request. Generated text is parsed as JSON and validated against the schema using the `jsonschema` crate. If invalid, generation is retried up to 3 times with increasing temperature to encourage diversity. More robust than the post-hoc `json_object` mode. Covered by `validate_accepts_valid_json_object`, `validate_rejects_invalid_json`, and `generate_with_schema_retries_until_valid` tests.
- [x] **Speculative decoding parity** — `speculative_generate` now supports prefix caching (restores cached K/V before priming), context shifting (`max_context`), and grammar constraints (`sample_constrained`). Previously only `generate_core` had these features; the speculative path was a second-class citizen.
- [x] **OpenAI-compatible streaming** — `POST /v1/chat/completions` with `stream: true` returns an SSE stream of `chat.completion.chunk` objects with incremental `delta.content`, ending with `finish_reason: "stop"` and `[DONE]`. Matches the OpenAI streaming protocol. Covered by `run_server_serves_openai_chat_stream` test.
- [x] **Stop sequences** — `GenerationConfig.stop` holds a list of strings that halt generation when any appear in the decoded output. Checked after each token in `generate_core` and `speculative_generate`; output is truncated to just before the matched sequence. Wired through `/chat`, `/complete`, and `/v1/chat/completions` request bodies as an optional `stop` array.
- [x] **EAGLE3 speculative decoding** — `src/draft.rs` introduces a `DraftModel` trait and an `MlpDraftModel` implementation: a tiny 2-layer MLP (embedding → FC1 + ReLU → FC2 → logits) that is orders of magnitude faster than the full transformer. The draft model loads from `draft.*` tensors in the runtime and falls back to borrowing the target model's embedding weights. `speculative_generate` uses the neural draft when present, otherwise falls back to the n-gram draft model. Wired into `AppState` so all serve endpoints (`/chat`, `/complete`, `/v1/chat/completions`) automatically benefit. Covered by `mlp_draft_model_produces_deterministic_tokens_when_greedy` and `neural_draft_model_speculative_matches_non_speculative` tests.
- [x] **OpenAI `/v1/completions` endpoint** — The legacy (non-chat) completions API: `POST /v1/completions { model, prompt, max_tokens, temperature, top_p, stop, stream }`. Returns `text_completion` objects with `choices[].text` (no chat template applied). Supports `stream: true` with SSE `text_completion` chunks, plus `logprobs`, `top_logprobs`, `grammar`, and `json_schema`. Standard across Ollama, llama.cpp server, vLLM, and SGLang. Covered by `run_server_serves_openai_completions` test.
- [x] **Concurrent transformer generation** — The prefix cache was a `Mutex` that serialized all transformer requests. Refactored to `RwLock` with brief lock hold times: read-lock during lookup, release during generation, write-lock during insertion. `generate_core` and `speculative_generate` now accept `CacheLookup` by value and return cacheable `KvCache`, letting callers manage their own locking. This allows multiple transformer requests to run in parallel (limited by CPU cores), dramatically improving throughput under load. Covered by `run_server_handles_concurrent_transformer_requests` test. True "dynamic batching" (grouping multiple sequences into a single forward pass) is primarily a GPU optimization; on CPU, concurrent single-sequence processing is more efficient.
- [x] **Attention allocation optimization** — `attention_kv` (`src/runtime/transformer.rs`) previously allocated a fresh `scores` Vec and called `softmax` (which allocated twice more) for every head, on every layer, on every token — ~3×n_heads allocations per attention call. For a 12-layer, 12-head model that's ~432 heap allocations per generated token. Rewrote `attention_kv` to use a single pre-allocated scratch buffer reused across heads, and added `softmax_inplace` which operates in-place with zero allocations. This eliminates the dominant allocation hotspot in the transformer forward pass. Full "Flash Attention" (fused softmax+matmul in tiled blocks to reduce memory bandwidth) is primarily a GPU optimization; on CPU, avoiding allocations and keeping computation cache-friendly provides comparable practical speedup. Covered by `softmax_inplace_matches_softmax` and `attention_kv_multi_head_no_alloc_bloat` tests.
- [x] **Mixed-precision KV cache** — `KvLayer::Mixed` (`src/runtime/transformer.rs`) stores recent tokens in FP32 and automatically moves older tokens to INT8 cold storage when the hot window (default 64 tokens) overflows. `k_all()` / `v_all()` concatenate cold (dequantized) + hot (FP32) in chronological order. This preserves accuracy for the actively attended context while reducing memory footprint vs pure FP32. Compatible with prefix caching, context shifting, and speculative decoding. Covered by `kv_layer_mixed_keeps_recent_in_fp32`, `kv_layer_mixed_matches_fp32_output`, and `kv_layer_mixed_shift_discards_oldest` tests.
- [x] **Anchor token preservation (StreamingLLM-style KV cache eviction)** — `GenerationConfig.anchor_tokens` (default 0) preserves the first N initial tokens during context shifting. Inspired by StreamingLLM, these "attention sink" tokens accumulate disproportionate attention scores; evicting them degrades generation quality. `KvLayer::shift_anchored` and `KvCache::shift_anchored` remove tokens from the middle (after anchors) rather than from the beginning. Works with all KV cache variants (FP32, INT8, Mixed). Covered by `kv_layer_fp32_shift_anchored_preserves_first`, `kv_layer_int8_shift_anchored_preserves_first`, `kv_layer_mixed_shift_anchored_preserves_first`, and `kv_layer_shift_anchored_falls_back_when_not_enough_tokens` tests. Set via `--anchor-tokens N` on `modelc run`.
- [x] **Request rate limiting / API key auth** — `modelc run` accepts `--api-key <key>` (requires `Authorization: Bearer <key>` on all endpoints except `/health` and `/info`) and `--rate-limit <N>` (max requests per minute per client IP). Both return standard HTTP status codes (`401 Unauthorized`, `429 Too Many Requests`). The middleware lives in `src/serve/auth.rs` using a per-IP token bucket with automatic refill. Covered by `run_server_api_key_rejects_unauthenticated`, `run_server_api_key_accepts_authenticated`, and `run_server_rate_limit_rejects_over_limit` tests.
- [x] **Reproducible generation via seed** — `GenerationConfig.seed: Option<u64>` enables deterministic sampling. The same prompt + seed always produces the same output (when `temperature > 0`). Wires through all endpoints (`/chat`, `/complete`, `/v1/chat/completions`, `/v1/completions`) and the CLI (`--seed`). Each generation step uses `StdRng::seed_from_u64(seed.wrapping_add(step))` so the sequence varies per position while remaining fully reproducible. Covered by `multinomial_same_seed_same_step_is_deterministic`, `multinomial_same_seed_different_step_differs`, and `sample_with_seed_is_deterministic` tests.
## Brainstorming (research-driven)
- [x] **Repetition penalty** — `GenerationConfig.repetition_penalty: f32` (default 1.0 = disabled). Applies standard transformers-style penalty: positive logits for seen tokens are divided by the penalty, negative logits are multiplied. This pushes repeated-token probabilities down when penalty > 1.0. Wires through all endpoints (`/chat`, `/complete`, `/v1/chat/completions`, `/v1/completions`) and the CLI (`--repetition-penalty`). Uses a bitmap (1024 u64s = 65536 bits) for O(1) seen-token lookup per vocab entry, keeping the per-step overhead minimal. Covered by `repetition_penalty_disabled_leaves_logits_unchanged`, `repetition_penalty_reduces_seen_token_logits`, `repetition_penalty_amplifies_negative_logits`, and `repetition_penalty_penalizes_each_seen_token_once` tests.
- [x] **Presence / frequency penalty** — `GenerationConfig.presence_penalty` and `frequency_penalty` (default 0.0 = disabled), the OpenAI-standard parameters that discourage token reuse. `apply_presence_frequency_penalty` (`src/generate.rs`) is additive on logits: presence subtracts once per seen token, frequency subtracts `count * penalty`. Wired through all text-generation endpoints (`/chat`, `/complete`, `/v1/chat/completions`, `/v1/completions`) as optional request fields and the CLI (`--presence-penalty`, `--frequency-penalty`). Applied alongside `repetition_penalty` in `sample_constrained`; both share a fast path that skips the per-token allocation when no penalties/constraints are active. Covered by `presence_frequency_penalty_disabled_is_noop`, `presence_penalty_subtracts_once_per_seen_token`, `frequency_penalty_scales_with_count`, `presence_and_frequency_combine`, and `presence_frequency_penalty_empty_tokens_is_noop` tests.
- [x] **Min-p sampling** — `GenerationConfig.min_p` (default 0.0 = disabled) keeps only tokens whose probability is at least `min_p` fraction of the maximum probability, then renormalizes. Simpler and often more effective than top-p (nucleus) because the threshold scales with the model's confidence per step. `apply_min_p` (`src/generate.rs`) runs in `sample` right after `apply_top_p`, so both can combine. Wired through all text-generation endpoints (`/chat`, `/complete`, `/v1/chat/completions`, `/v1/completions`) as an optional `min_p` request field and the CLI (`--min-p`). Covered by `apply_min_p_drops_low_probability_tokens`, `apply_min_p_keeps_tokens_above_threshold`, and `apply_min_p_with_zero_is_noop` tests.
- [x] **Request cancellation on disconnect** — When an SSE streaming client disconnects, generation now aborts instead of running to completion. A cooperative `cancel: Option<Arc<AtomicBool>>` on `GenerationConfig` (`src/generate.rs`) is checked once per token in `generate_core` and `speculative_generate`; the SSE response is wrapped in `CancelOnDrop` (`src/serve/mod.rs`), whose `Drop` trips the flag when axum drops the response body (client gone). Wired into all three streaming paths (`/chat/stream`, `/v1/chat/completions` stream, `/v1/completions` stream); drip loops also stop sending once the channel closes. Non-streaming endpoints still compute to completion (a single unit of work with no incremental send to observe a disconnect). Covered by `cancel_flag_defaults_to_not_cancelled`, `cancel_flag_set_is_cancelled`, `cancel_on_drop_sets_flag`, and `cancellation_flag_stops_generation` tests.
- [x] **Tokenize endpoint** — `POST /tokenize` (or `/v1/tokenize`) returns token IDs and count for a given prompt. Standard in Ollama and llama.cpp server; useful for prompt management.
- [~] **Transformer dynamic batching** — **Deferred (by design).** True dynamic batching groups multiple concurrent sequences into a single batched forward pass. Implementing it here requires parallel batched implementations of every transformer primitive (`linear_with`, `layer_norm`, `rms_norm`, `gelu`/`silu`, `rope`, `attention_kv`) plus batched KV-cache handling — a large rewrite that must stay bit-identical to the codegen-mirrored single-sequence path. The codebase already ships the CPU-appropriate alternative: **concurrent single-sequence processing** (`run_server_handles_concurrent_transformer_requests`) driven by an `RwLock` prefix cache, which the existing assessment notes is more efficient than dynamic batching on CPU (transformer generation is memory-bandwidth-bound; per-core concurrent single-sequence already saturates available bandwidth without the coordination overhead of a continuous-batching scheduler). Dynamic batching becomes worthwhile only with a batched-matmul **GPU** backend; modelc's Metal kernels are matmul/elementwise, not a batched-transformer runtime. Revisit when a batched GPU forward path exists. Tracking this as deferred rather than implemented to keep the feature honest.
- [x] **System / hardware info endpoint** — `GET /v1/system` exposes CPU/GPU memory, core count, Metal availability. Useful for orchestration and debugging.
- [x] **Model quantization preview** — `modelc inspect --quant-sizes` shows what artifact sizes would be for FP32, FP16, INT8, INT4, Q4_0 without actually quantizing. Computed from element counts (shape-based) via `QuantPreview` (`src/model.rs`), so it's accurate regardless of the tensors' current dtype. `print_quant_sizes` (`src/compiler.rs`) renders a format/size/savings table. Q4_0 uses per-tensor block rounding (18 bytes / 32 elements). Covered by `quant_preview_bytes_per_format`, `quant_preview_q4_0_rounds_up_per_block`, `quant_preview_int4_rounds_up_odd`, `preview_size_bytes_matches_dtype_scaling`, `preview_size_bytes_is_shape_based_not_storage_based`, `all_formats_returns_expected_set`, and `quant_sizes_preview_runs_without_error` tests.
## Brainstorming (competitive intelligence — Ollama / llama.cpp server / vLLM / SGLang / TGI)
- [x] **Graceful shutdown on signals** — `modelc run` catches `SIGINT`/`SIGTERM` via `tokio::signal`, stops accepting new connections with `axum::serve(..).with_graceful_shutdown(..)`, and drains in-flight requests (including SSE streams via `CancelOnDrop`) before exiting. Compiled binaries generated by `modelc compile` also include `with_graceful_shutdown`.
- [x] **Max-concurrent-request backpressure** — `modelc run --max-concurrent N` limits in-flight inference requests with a `tokio::sync::Semaphore`. Excess requests receive `503 Service Unavailable`. Exempts `/health`, `/info`, and `/metrics` so probes and observability still work under load. Builds on the existing `metrics::ActiveRequestGuard`.
- [x] **`/v1/embeddings` OpenAI parity** — `POST /v1/embeddings` returns OpenAI-compatible embeddings with `dimensions` (embedding length) and honors `encoding_format` (`"float"` / `"base64"`). Float arrays and base64-encoded little-endian bytes are both supported. Includes `usage` with `prompt_tokens` and `total_tokens`. Covered by `run_server_serves_openai_v1_embeddings` test.
- [x] **`/api/version` / server version endpoint** — `GET /api/version` exposes `{ version, git_sha }` from `build.rs` (`MODELC_GIT_SHA`) for orchestration and health checks. Covered by `run_server_serves_api_version` test.
## Brainstorming (competitive intelligence — July 2026)
- [x] **`modelc cp <source> <dest>`** — Duplicate a model in the local store. Ollama, Docker, and every model manager supports copying. `copy_model` (`src/store.rs`) resolves the source (name or path), refuses to overwrite an existing destination, and copies the `.modelc` artifact. Covered by `test_copy_model_creates_copy`, `test_copy_model_rejects_existing_dest`, and `test_copy_model_rejects_missing_source` tests.
- [x] **Prompt lookup decoding** — `PromptLookupDraftModel` (`src/draft.rs`) implements a weight-free draft model for speculative decoding. Instead of a separate neural network, it searches the current token context for the most recent occurrence of the last N tokens and proposes the continuation that followed. Particularly effective for code completion, document Q&A, and any task where output tends to repeat substrings from the prompt. Used as the default draft model when no neural draft weights (`draft.*`) are present, so all serve endpoints automatically benefit. Covered by `prompt_lookup_finds_continuation`, `prompt_lookup_returns_empty_when_no_match`, `prompt_lookup_returns_empty_for_short_context`, `prompt_lookup_caps_at_gamma`, `prompt_lookup_finds_most_recent_match`, and `prompt_lookup_zero_gamma_returns_empty` tests.
- [x] **Logit bias** — `GenerationConfig.logit_bias` (`HashMap<u32, f32>`) implements the OpenAI-standard `logit_bias` parameter. Positive values boost a token's likelihood; negative values suppress it. Applied additively to logits after penalties but before the grammar constraint mask. Wired through all text-generation endpoints (`/chat`, `/complete`, `/v1/chat/completions`, `/v1/completions`) as an optional `logit_bias` request field (JSON object mapping token ID to bias). Empty map = fast path (no allocation). Covered by `logit_bias_boosts_token_in_greedy`, `logit_bias_suppresses_token_in_greedy`, and `logit_bias_empty_is_fast_path` tests.
- [x] **`echo` parameter for `/v1/completions`** — OpenAI-standard `echo: true` prepends the prompt text to the completion in the response. In streaming mode, the prompt is emitted as the first SSE chunk. Covered by `run_server_completions_echo` test.
- [x] **`n` parameter for `/v1/completions`** — OpenAI-standard `n` parameter generates N independent completions (clamped to 1–20). The response includes N `choices` entries with incrementing `index`. `completion_tokens` in `usage` is the sum across all choices. Covered by `run_server_completions_n` test.
- [x] **`/props` endpoint** — `GET /props` (llama.cpp-compatible) exposes server properties: model name, architecture, total params/bytes, chat template, and default generation parameters (`max_tokens`, `temperature`, `top_p`, `min_p`, `repetition_penalty`, `presence_penalty`, `frequency_penalty`, `gamma`). Exempt from auth and backpressure so clients can discover server capabilities without credentials. Covered by `run_server_serves_props` test.
- [x] **`/detokenize` endpoint** — `POST /detokenize` (llama.cpp-compatible) decodes token IDs back to text using the model's byte-level BPE tokenizer. Complements `/tokenize` for round-trip token management. Accepts `{ "tokens": [id, ...] }`, returns `{ "text": "..." }`. Covered by `run_server_detokenize` test.
- [x] **`suffix` parameter for `/v1/completions`** — OpenAI-standard `suffix` string is appended after the generated completion in the response. When combined with `echo: true`, the output format is `prompt + completion + suffix`. Covered by `run_server_completions_suffix` test.
- [x] **`n` parameter for `/v1/chat/completions`** — OpenAI-standard `n` parameter generates N independent chat completions (clamped to 1–20). The response includes N `choices` entries with incrementing `index` and `message` fields. `completion_tokens` in `usage` is the sum across all choices. Covered by `run_server_chat_completions_n` test.
- [x] **`stream_options.include_usage` for `/v1/chat/completions`** — OpenAI-standard streaming option. When `stream_options: { include_usage: true }` is set, the SSE stream emits a final chunk with empty `choices` and a `usage` object (`prompt_tokens`, `completion_tokens`, `total_tokens`) before `[DONE]`. Covered by `run_server_chat_stream_include_usage` test.
- [x] **`stream_options.include_usage` for `/v1/completions`** — Same OpenAI-standard streaming option, now also on the legacy completions endpoint. When `stream_options: { include_usage: true }` is set, the SSE stream emits a final `text_completion` chunk with empty `choices` and a `usage` object before `[DONE]`. Covered by `run_server_completions_stream_include_usage` test.
- [x] **`GET /v1/models/:id`** — OpenAI-standard model retrieval endpoint. Returns a single model object (`{ id, object: "model", created, owned_by }`) when the ID matches the loaded model. Returns `404` with `model_not_found` error for unknown IDs. Covered by `run_server_retrieve_model` test.
- [x] **`best_of` parameter for `/v1/completions`** — OpenAI-standard `best_of` generates N completions server-side (clamped to 1–20), sorts by token count (proxy for highest logprob), and returns the top `n`. When `best_of > n`, the client receives only the best `n` choices. Cannot be used with `stream`. Covered by `run_server_completions_best_of` test.
- [x] **`user` field on both endpoints** — OpenAI-standard `user` parameter accepted on `/v1/chat/completions` and `/v1/completions` for abuse tracking. Accepted but ignored (no multi-tenant state). Covered by `run_server_chat_with_user_field` test.
- [x] **`response_format: { type: "json_schema" }` for `/v1/chat/completions`** — OpenAI-standard Structured Outputs. When `response_format.type` is `"json_schema"`, the server injects the schema as a system prompt and validates/retries generated output using `generate_with_schema` (up to 3 retries with increasing temperature). Covered by `run_server_chat_response_format_json_schema` test.
- [x] **`tool_choice` parameter for `/v1/chat/completions`** — OpenAI-standard tool choice control. Supports `"auto"` (default, model decides), `"none"` (tools are not offered), `"required"` (model must call a tool), and `{"type": "function", "function": {"name": "..."}}` (model must call the named tool). When `"none"`, tool descriptions are not injected and `tool_calls` is never returned. Covered by `run_server_chat_tool_choice_none` and `run_server_chat_tool_choice_required` tests.
- [x] **`name` field on `ChatMessage`** — OpenAI-standard `name` field on message objects (used to identify the speaker in multi-user conversations). Accepted and serialized back in responses. Covered by `run_server_chat_with_name_field` test.
- [x] **`max_completion_tokens` on both endpoints** — OpenAI's newer parameter name for `max_tokens` (introduced for o1+ models). Accepted on `/v1/chat/completions` and `/v1/completions`. When both `max_completion_tokens` and `max_tokens` are provided, `max_completion_tokens` takes precedence. Covered by `run_server_chat_max_completion_tokens` test.
- [x] **Batch `prompt` (array of strings) for `/v1/completions`** — OpenAI-standard batch mode. When `prompt` is an array of strings, the server generates completions for each prompt and returns them as separate `choices` with incrementing `index`. `prompt_tokens` in `usage` is the sum across all prompts. Covered by `run_server_completions_batch_prompt` test.
## Audit & optimization pass (July 2026)
- [x] **Dead-code removal** — Deleted `src/main.rs` (476 lines): a stale duplicate of the active `src/bin/modelc.rs` binary entry that was never compiled (the `[[bin]]` target points at `src/bin/modelc.rs`; verified via `cargo build --message-format=json`). Updated `ARCHITECTURE.md` to point at the real entry.
- [x] **Clippy / warning clean** — Fixed all clippy lints and dead-code warnings: `collapsible_match` (gguf chat-template), `manual_clamp` → `clamp` (×2 in openai `n`), `unnecessary_sort_by` → `sort_by_key(Reverse)` (best_of), `redundant_field_names` (usage struct), and silenced intentionally-ignored OpenAI fields (`JsonSchemaDef.name`/`.strict`, `ChatCompletionRequest.user`, `CompletionRequest.user`) with `#[allow(dead_code)]`. `cargo clippy --all-targets -- -D warnings` is now clean.
- [x] **Release profile tuning** — Added `[profile.release]` (`opt-level = 3`, `lto = "thin"`, `codegen-units = 1`, `strip = "symbols"`). Release binary drops to ~9.4 MiB (vs 46 MiB debug). Skipped `panic = "abort"` so `cargo test` and the Tokio runtime keep unwinding.
- [x] **Memoized runtime structural view** — `forward_gpt2_cached` / `forward_llama_cached` / `embed_*` previously rebuilt a synthetic `Model` (cloning every tensor name + shape into a fresh `HashMap`) on **every generated token** just to feed the `arch` layer/hidden/head inspection. `Runtime` now memoizes this view in a `OnceLock<Model>` (`Runtime::model_view`); structure is built once per `Runtime` and free on every subsequent token. Safe because `Runtime` is only ever replaced wholesale (LoRA load/unload via `*runtime = Runtime::from_raw(...)`), never mutated in place — a swapped-in `Runtime` gets a fresh lock. Turns O(N·T) name-clones over an N-token generation pass into O(T + N). Covered by `model_view_is_memoized_and_exposes_shapes` and `model_view_refreshes_after_runtime_swap` tests.
- [x] **Zero-copy FP32 KV attention path** — `gpt2_attention` / `llama_attention` called `KvLayer::k_all()` / `v_all()` on every token, which `.clone()`s the *entire* accumulated FP32 K/V buffer just to feed `attention_kv` a `&[f32]` — an O(seq_len·hidden) copy per layer per token that grows with context. Added `KvLayer::k_all_cow` / `v_all_cow` returning `Cow<[f32]>`: the FP32 default path borrows the backing buffer directly (zero-copy), while quantized (INT8/Mixed) variants still materialize owned FP32 as before. Attention now reads K/V straight from the cache for the common case. Covered by `kv_layer_fp32_cow_is_zero_copy_and_correct`, `kv_layer_int8_cow_is_owned`, and `attention_kv_cow_path_matches_materialized` tests.
- [x] **SIMD GEMV, runtime + codegen (parity preserved)** — The autoregressive hot path was a scalar FP reduction (`acc += w*x`) that the compiler cannot autovectorize without fast-math, and `ops::matmul`'s AVX/NEON kernels vectorize over the N dimension — useless for GEMV (N=1). Added `src/runtime/gemv.rs` (`gemv_dot`): a K-dimension vectorized dot product (NEON f32x4 FMA on aarch64, AVX f32x8 mul+add on x86_64, scalar fallback elsewhere) with a fixed horizontal-reduction order and scalar tail. To preserve the documented `modelc run` ↔ `modelc compile` bit-identical invariant, the kernel is a **single source** compiled into the runtime *and* emitted verbatim into generated binaries via `include_str!` in `codegen/native/helpers.rs` (`//!` module-doc lines stripped at emit time since they'd be invalid mid-file). Wired into runtime `linear_with` (transformer) and `gemv_bias` / `batched_gemv_bias` (serve infer), and into the emitted `matmul_bias` (MLP) and `linear` (GPT-2 / LLaMA) helpers. The runtime `None`-bias path returns the dot verbatim (no `0.0 + dot`) so it matches the emitted bias-free `linear` exactly. Covered by `gemv_dot_*` unit tests, `emitted_helpers_share_simd_gemv_kernel` (structural wiring), and the previously-ignored `codegen_{mlp,gpt2,llama}_emitted_project_compiles` tests (now also exercise the embedded kernel — all three pass).