# TODO — Plan
## Implemented
- [x] **Parsers** — GGUF (F32/F16/BF16/I8/I16/I32/I64/F64, Q4_0/Q5_0/Q8_0/Q4_K/Q6_K dequant), ONNX (inline/external/segmented initializers), PyTorch (Safetensors-in-ZIP), Safetensors (`src/parsers/`).
- [x] **ONNX graph execution** — parses ops (MatMul, Gemm, Add, Mul, Div, Sub, Relu, Softmax, LayerNorm, Transpose, Reshape, Sigmoid, Tanh, Identity, Cast) into a JSON execution plan stored in metadata (`src/onnx_exec/`).
- [x] **MLP codegen forward** — FP32 GEMV chain when `architecture == "mlp"` (`src/codegen/native/`).
- [x] **GPT2 / LLaMA codegen skeletons** — simplified transformer forward pass stubs (`src/codegen/native/forward.rs`).
- [x] **Auto architecture inference** — detects `gpt2`, `llama`, `bert`, `mlp` from tensor naming patterns (`src/model.rs`).
- [x] **Compile peak RAM** — Embedding writes stream via `BufWriter` (no auxiliary Vec blob).
- [x] **crates.io checklist** — README subsection with publish steps.
- [x] **Single-file artifact format** — `.modelc` binary with JSON header + raw tensor blob (`src/pack.rs`).
- [x] **Cross-platform runtime** — Platform-specific store paths via `dirs` crate; `#[cfg]` Unix/Windows guards.
- [x] **Apple Silicon Metal acceleration** — Full kernels for relu, add, mul_scalar, softmax, layer_norm (`src/metal.rs`, `src/compute/shaders.metal`).
- [x] **Ollama-like UX** — `pack`, `run`, `pull`, `list` with local model store (`src/store.rs`).
- [x] **Size optimization** — `--compress` flag on `pack` uses zstd (version 2 format).
- [x] **Quantization at pack time** — `--quantize fp16/int8/int4` converts F32 tensors; INT4/INT8 auto-dequantized on `run`.
- [x] **Weight pruning** — `--prune <threshold>` zeroes small weights.
- [x] **CPU SIMD matmul** — AVX (x86_64) and NEON (aarch64) paths with runtime feature detection.
- [x] **Multi-threaded CPU ops** — `rayon` parallelization for large matmuls.
- [x] **GPU memory pressure handling** — pre-allocation checks before Metal buffer creation.
- [x] **Store metadata enrichment** — `modelc list` reads artifact headers to show architecture, params, and compression status.
- [x] **Remote model pull** — `modelc pull <url>` supports HTTP/HTTPS downloads with versioning.
- [x] **Benchmark command** — `modelc bench <artifact>` measures warm/cold latency and throughput.
- [x] **Artifact verification** — `modelc verify <artifact>` checks header integrity.
- [x] **Export to Safetensors** — `modelc export <artifact> -o out.safetensors`.
- [x] **Model search** — `modelc search <query>` filters store by architecture, size, or name.
- [x] **Config file support** — `~/.modelc/config.toml` for default bind, store path, compression.
- [x] **Shell completions** — bash/zsh/fish via `modelc completions <shell>`.
- [x] **Per-op profiling** — `--profile` flag on `run` prints per-step timing.
- [x] **Model versioning** — store multiple versions and `modelc switch <name> <version>`.
- [x] **Model card generation** — `modelc inspect --readme` generates Markdown model card.
- [x] **Docker/OCI image generation** — `modelc containerize <artifact>` emits Dockerfile + entrypoint.
- [x] **LoRA adapter support** — load and apply LoRA adapters at runtime (`src/lora.rs`).
- [x] **Chat/completion endpoints** — `/chat`, `/complete`, `/chat/stream` (SSE).
- [x] **Batch inference API** — `/infer` accepts `{"inputs": [...]}`.
- [x] **Code modularization** — split large files into focused submodules (`gguf/`, `onnx/`, `onnx_exec/`, `codegen/native/`, `serve/`).
- [x] **Transformer runtime in `run`** — `modelc run` now executes GPT-2 / LLaMA forward passes directly from the in-memory runtime (`src/runtime/transformer.rs`) instead of echoing input. Structure inspection shared with codegen via `src/arch.rs`; numeric helpers mirror the emitted codegen so `run` and `compile` outputs match.
## Next / Competitive gaps
- [x] **`modelc rm <name>`** — Delete a model from the local store (and its versioned copies). Every model manager (Ollama, Docker) has this; we don't.
- [x] **OpenAI-compatible API** — `/v1/models`, `/v1/chat/completions` (non-streaming). Ollama, llama.cpp server, vLLM, and SGLang all expose this; it's the dominant integration pattern for client libraries.
- [x] **`/embeddings` endpoint** — Produce vector embeddings from the hidden-state pooler via `POST /embeddings`.
- [x] **KV cache in transformer runtime** — `KvCache` struct stores per-layer K/V vectors. `forward_gpt2_cached` and `forward_llama_cached` accept optional cache, append current K/V, and compute attention over all cached positions via `attention_kv`. Eliminates redundant K/V recomputation during autoregressive generation.
- [x] **Built-in tokenization** — Byte-level BPE tokenizer (`src/tokenizer.rs`) with encode/decode and greedy merge algorithm. Foundation for real text-in/text-out transformer inference.
- [x] **GGUF tokenizer metadata extraction** — `extract_tokenizer_metadata` reads vocab, merges, BOS/EOS IDs from GGUF KV metadata. `BpeTokenizer::from_gguf` constructs a working tokenizer from this data.
- [x] **`/health` endpoint** — Standard liveness probe for inference servers and container orchestration.
- [x] **Model deletion guard** — Optional `--force` flag on `rm`; refuse to delete the main artifact if versioned copies exist unless `--all` or `--force` is provided.
## Competitive gaps (research-driven)
- [x] **Autoregressive text generation** — `src/generate.rs` provides `generate()` with greedy and temperature sampling. Wires into `run_text_inference` for transformer models so `/chat`, `/complete`, and `/v1/chat/completions` return real generated text instead of logits JSON.
- [x] **Chat template support** — reads Jinja2 chat templates from GGUF metadata (`tokenizer.chat_template`) and applies them to format messages before tokenization via `minijinja`. Falls back to simple `role: content\n` concatenation when no template is present.
- [x] **GGUF quantization inference** — GGUF parser preserves Q4_0, Q5_0, Q8_0, Q4_K, Q6_K block-quantized bytes in IR instead of expanding to F32 at parse time. `Runtime::from_raw` dequantizes on-the-fly via `dequantize_gguf_tensor`. This reduces `.modelc` artifact size (~8x for Q4_0) while keeping inference functional. Codegen path calls `dequantize_in_place` before generating the server.
- [x] **Structured output / JSON mode** — `response_format: { type: "json_object" }` on `/v1/chat/completions` injects a system prompt instructing JSON-only output, then post-processes generated text with `extract_json_object` to extract the first well-formed JSON object or array. Falls back to raw text if no valid JSON is found.
- [x] **Function calling / tool use** — `POST /v1/chat/completions` accepts an OpenAI-compatible `tools` array. When tools are present, a system prompt describing available tools is injected. Generated output is parsed for a `tool_calls` JSON array; if found, the response returns `tool_calls` with `finish_reason: "tool_calls"`. Otherwise falls back to regular content.
- [x] **Continuous batching (MLP)** — `POST /infer` with multiple `inputs` and MLP architecture now routes to `run_mlp_forward_batched`, which computes all items in a single pass via `batched_gemv_bias`. Eliminates redundant weight traversal and improves cache locality. Transformer generation batching remains future work.
- [x] **Speculative decoding** — `generate()` supports `config.gamma > 0` to enable speculative decoding with an n-gram draft model (`src/generate.rs`). The draft model proposes `gamma` candidate tokens by looking up trigram continuations from the prefix; the target model verifies each candidate in a loop using the KV cache. Accepted tokens skip the sampling step. Establishes infrastructure for future faster draft models (e.g., smaller transformer or prompt lookup decoding).
- [x] **Top-p / nucleus sampling** — `GenerationConfig.top_p` enables nucleus filtering: probabilities are sorted descending, the smallest prefix whose cumulative sum exceeds `top_p` is kept, and the rest are zeroed before renormalization and sampling. Wired through all text generation endpoints (`/chat`, `/complete`, `/v1/chat/completions`) as an optional request parameter.
- [x] **Streaming token generation** — `/chat/stream` now generates token IDs via `generate_token_ids()`, then decodes and emits each token incrementally through SSE. After each new token, the cumulative sequence is decoded and only the text delta (newly added prefix) is sent. `done: true` marks the final chunk. Non-transformer models fall back to a single chunk.
- [x] **Batched embeddings** — `POST /embeddings` now accepts `inputs: ["...", "..."]` and returns `embeddings: [{"embedding": [...], "index": 0}, ...]`. Single-input `input: "..."` remains backward compatible and returns `embedding: [...]`.
- [x] **Token-level logprobs** — `POST /v1/chat/completions` accepts OpenAI-compatible `logprobs: true` and `top_logprobs: N` (0–20). `generate_with_logprobs()` (`src/generate.rs`) drives the autoregressive loop and records `ln(P(token))` from the raw (pre-temperature) softmax plus up to N top alternatives per position. The response carries `choices[].logprobs.content` with per-token `token`, `logprob`, raw UTF-8 `bytes`, and `top_logprobs` entries. Non-transformer models return an empty `content` array; when `logprobs` is omitted the field serializes to `null` (matching OpenAI).
- [x] **Quantized KV cache** — `KvLayer::Int8` (`src/runtime/transformer.rs`) stores per-token K/V as INT8 with per-token scales (4× memory reduction vs FP32). `KvCache::new_quantized(n_layers, hidden, use_int8)` selects INT8 vs FP32 at creation; `GenerationConfig.use_int8_kv` wires the choice through `generate_token_ids`, `generate_with_logprobs`, and `speculative_generate`. Dequantization is transparent via `k_all`/`v_all`. Covered by `kv_layer_int8_roundtrip` / `kv_layer_int8_multiple_tokens` tests.
## Brainstorming (research-driven)
- [x] **Prefix caching** — `PrefixCache` (`src/prefix_cache.rs`) stores `KvCache` snapshots keyed by token sequence. Longest-prefix matching lets later requests whose prompt starts with a cached sequence restore the saved K/V and only process the divergent suffix. Wired into `generate_core`, `generate_token_ids_with_cache`, and `generate_with_logprobs_with_cache`. Serve layer (`run_text_inference_with_config`, `run_text_inference_token_ids`, `run_text_inference_with_logprobs`) locks the per-state cache and passes it through. LRU eviction bounds memory to 32 entries.
- [x] **LoRA loading at runtime via HTTP** — `POST /lora/load { path: "...", alpha: 1.0 }` clones the base model tensors, applies a Safetensors LoRA adapter (`src/lora.rs`), and swaps the runtime atomically via `RwLock<Runtime>`. `POST /lora/unload` restores the base model from the stored `base_tensors`. No server restart required. Covered by `run_server_lora_load_bad_path_returns_error` and `run_server_lora_unload_restores_base` tests.
- [x] **Grammar-based constrained decoding** — `GenerationConfig.constraint` holds an optional `Arc<dyn Constraint>`. `RegexConstraint` (`src/constraint.rs`) wraps a regex and masks invalid tokens to `-inf` before sampling. The `sample_constrained` helper in `generate.rs` decodes the partial output, builds a per-token validity mask, and applies it. HTTP endpoints (`/chat`, `/complete`, `/v1/chat/completions`) accept an optional `grammar` string (regex pattern) per request. The heuristic tries common suffixes to avoid false negatives; it is permissive rather than exact, which is acceptable for a minimal implementation.
- [x] **Prometheus metrics endpoint** — `GET /metrics` exposes `modelc_requests_total`, `modelc_inference_duration_seconds` (histogram with buckets 0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, +Inf), `modelc_active_requests` gauge, and `modelc_tokens_generated_total` counter. Handlers use `ActiveRequestGuard` and `InferenceTimer` RAII structs to track automatically. Rendered in Prometheus text exposition format. Covered by `run_server_metrics_returns_prometheus_text` test.
- [x] **Context shifting (sliding window KV cache)** — `GenerationConfig.max_context` sets a hard context limit. When total tokens exceed it, `generate_core` shifts both the `KvCache` (`KvLayer::shift` removes oldest K/V vectors) and `token_ids` left, keeping roughly the newest 75%. `prompt_len` is adjusted so generation continues seamlessly. Covered by `kv_layer_fp32_shift_discards_oldest`, `kv_layer_int8_shift_discards_oldest`, and `kv_cache_shift_syncs_all_layers` tests.
- [x] **JSON Schema constrained generation** — `POST /chat`, `/complete`, and `/v1/chat/completions` accept an optional `json_schema` object per request. Generated text is parsed as JSON and validated against the schema using the `jsonschema` crate. If invalid, generation is retried up to 3 times with increasing temperature to encourage diversity. More robust than the post-hoc `json_object` mode. Covered by `validate_accepts_valid_json_object`, `validate_rejects_invalid_json`, and `generate_with_schema_retries_until_valid` tests.
- [x] **Speculative decoding parity** — `speculative_generate` now supports prefix caching (restores cached K/V before priming), context shifting (`max_context`), and grammar constraints (`sample_constrained`). Previously only `generate_core` had these features; the speculative path was a second-class citizen.
- [x] **OpenAI-compatible streaming** — `POST /v1/chat/completions` with `stream: true` returns an SSE stream of `chat.completion.chunk` objects with incremental `delta.content`, ending with `finish_reason: "stop"` and `[DONE]`. Matches the OpenAI streaming protocol. Covered by `run_server_serves_openai_chat_stream` test.
- [x] **Stop sequences** — `GenerationConfig.stop` holds a list of strings that halt generation when any appear in the decoded output. Checked after each token in `generate_core` and `speculative_generate`; output is truncated to just before the matched sequence. Wired through `/chat`, `/complete`, and `/v1/chat/completions` request bodies as an optional `stop` array.
- [x] **EAGLE3 speculative decoding** — `src/draft.rs` introduces a `DraftModel` trait and an `MlpDraftModel` implementation: a tiny 2-layer MLP (embedding → FC1 + ReLU → FC2 → logits) that is orders of magnitude faster than the full transformer. The draft model loads from `draft.*` tensors in the runtime and falls back to borrowing the target model's embedding weights. `speculative_generate` uses the neural draft when present, otherwise falls back to the n-gram draft model. Wired into `AppState` so all serve endpoints (`/chat`, `/complete`, `/v1/chat/completions`) automatically benefit. Covered by `mlp_draft_model_produces_deterministic_tokens_when_greedy` and `neural_draft_model_speculative_matches_non_speculative` tests.
- [x] **OpenAI `/v1/completions` endpoint** — The legacy (non-chat) completions API: `POST /v1/completions { model, prompt, max_tokens, temperature, top_p, stop, stream }`. Returns `text_completion` objects with `choices[].text` (no chat template applied). Supports `stream: true` with SSE `text_completion` chunks, plus `logprobs`, `top_logprobs`, `grammar`, and `json_schema`. Standard across Ollama, llama.cpp server, vLLM, and SGLang. Covered by `run_server_serves_openai_completions` test.
- [x] **Concurrent transformer generation** — The prefix cache was a `Mutex` that serialized all transformer requests. Refactored to `RwLock` with brief lock hold times: read-lock during lookup, release during generation, write-lock during insertion. `generate_core` and `speculative_generate` now accept `CacheLookup` by value and return cacheable `KvCache`, letting callers manage their own locking. This allows multiple transformer requests to run in parallel (limited by CPU cores), dramatically improving throughput under load. Covered by `run_server_handles_concurrent_transformer_requests` test. True "dynamic batching" (grouping multiple sequences into a single forward pass) is primarily a GPU optimization; on CPU, concurrent single-sequence processing is more efficient.
- [x] **Attention allocation optimization** — `attention_kv` (`src/runtime/transformer.rs`) previously allocated a fresh `scores` Vec and called `softmax` (which allocated twice more) for every head, on every layer, on every token — ~3×n_heads allocations per attention call. For a 12-layer, 12-head model that's ~432 heap allocations per generated token. Rewrote `attention_kv` to use a single pre-allocated scratch buffer reused across heads, and added `softmax_inplace` which operates in-place with zero allocations. This eliminates the dominant allocation hotspot in the transformer forward pass. Full "Flash Attention" (fused softmax+matmul in tiled blocks to reduce memory bandwidth) is primarily a GPU optimization; on CPU, avoiding allocations and keeping computation cache-friendly provides comparable practical speedup. Covered by `softmax_inplace_matches_softmax` and `attention_kv_multi_head_no_alloc_bloat` tests.
- [x] **Mixed-precision KV cache** — `KvLayer::Mixed` (`src/runtime/transformer.rs`) stores recent tokens in FP32 and automatically moves older tokens to INT8 cold storage when the hot window (default 64 tokens) overflows. `k_all()` / `v_all()` concatenate cold (dequantized) + hot (FP32) in chronological order. This preserves accuracy for the actively attended context while reducing memory footprint vs pure FP32. Compatible with prefix caching, context shifting, and speculative decoding. Covered by `kv_layer_mixed_keeps_recent_in_fp32`, `kv_layer_mixed_matches_fp32_output`, and `kv_layer_mixed_shift_discards_oldest` tests.
- [x] **Anchor token preservation (StreamingLLM-style KV cache eviction)** — `GenerationConfig.anchor_tokens` (default 0) preserves the first N initial tokens during context shifting. Inspired by StreamingLLM, these "attention sink" tokens accumulate disproportionate attention scores; evicting them degrades generation quality. `KvLayer::shift_anchored` and `KvCache::shift_anchored` remove tokens from the middle (after anchors) rather than from the beginning. Works with all KV cache variants (FP32, INT8, Mixed). Covered by `kv_layer_fp32_shift_anchored_preserves_first`, `kv_layer_int8_shift_anchored_preserves_first`, `kv_layer_mixed_shift_anchored_preserves_first`, and `kv_layer_shift_anchored_falls_back_when_not_enough_tokens` tests. Set via `--anchor-tokens N` on `modelc run`.
- [x] **Request rate limiting / API key auth** — `modelc run` accepts `--api-key <key>` (requires `Authorization: Bearer <key>` on all endpoints except `/health` and `/info`) and `--rate-limit <N>` (max requests per minute per client IP). Both return standard HTTP status codes (`401 Unauthorized`, `429 Too Many Requests`). The middleware lives in `src/serve/auth.rs` using a per-IP token bucket with automatic refill. Covered by `run_server_api_key_rejects_unauthenticated`, `run_server_api_key_accepts_authenticated`, and `run_server_rate_limit_rejects_over_limit` tests.