# Changelog
## 0.6.0 (2026-10-17)
- **Resident int8 vector index.** New `ResidentIndex` (`src/resident.rs`) and `VectorStore::resident_index(model_id)`: one read of a model's vectors into memory, each row quantised to `i8` with one `f32` scale (`absmax / 127`). `ResidentIndex::nearest(query, limit)` scores by cosine on the dequantised rows and orders ties by key, like `VectorStore::nearest`, without a SQLite scan per query. It is a snapshot: later writes are not seen. Written, unverified: not compiled or run.
- **Reciprocal-rank fusion.** New `reciprocal_rank_fusion(&[&[K]], k)` and `RRF_K` (60) in `src/fusion.rs`: sums `1 / (k + rank)` per item over ranked lists, returns best first, ties in first-appearance order. No feature gate and no dependency. Written, unverified.
- **Model bundle files.** New `bundle_files(dir) -> Result<Option<Vec<String>>, EmbedError>` lists the files the Core ML, E40 or Q4 layout requires and checks each exists (`MissingArtifact` otherwise); `None` when `dir` is none of them. Written, unverified.
- **Installed model directory.** New `installed_model_dir(exe, subdir) -> Option<PathBuf>`: the resources directory of `rightkit_onnx::installed_resource_dir(exe)` joined with a single plain `subdir` name. Returns the candidate without an existence check. Requires `rightkit-onnx` 0.2.6 (see the crate's `Cargo.toml`, whose requirement is still `0.2.2`). Written, unverified.
- No change to existing behaviour or public types.
## 0.6.0
- **Flat schema-2 bundle manifest (E40).** A bundle with a top-level `manifest.json` (`"schema": 2`) names its graphs, weights and token table: `graphs.query` (batch 1, lengths 16 to 512, a `file` template such as `model_{L}.onnx`), `graphs.ingest` (entries of batch 4 and 8, lengths 32, 128, 512, templates such as `ingest-b8_{L}.onnx`), `weights` (the one `model.onnx_data` every graph references; weights are never copied) and `token_table`. The manifest is parsed strictly: an unknown or missing schema version, a malformed entry, a template without `{L}`, a file name with a path part, an unknown token-table format, a `dtype_out` other than float16, a duplicate ingest batch or any listed file that is missing fails `load`. A bundle with no manifest is the legacy layout (`model_<L>.onnx` for 32 to 512, fp16 table, no ingest graphs) and loads unchanged. `pipeline_id()` and the parity cache key now hash every file the manifest names (manifest, query and ingest graphs, weights, table files, tokenizer).
- **Batched ingest on DirectML.** `Embedder::embed_batch` sorts documents by length and takes chunks of the remaining ones: the batch-8 ingest graph for five or more, batch 4 for two to four, the batch-1 query graph for one (only batches the bundle ships within the configured cap). A chunk runs on the smallest length that holds its longest text (a chunk longer than the longest ingest graph runs row by row on the query graphs). A short chunk is filled with rows of attention mask 0 (pad-token embeddings) and those rows are dropped from the output: no embedding comes back for padding. Results are in input order. Queries (`embed_query`, `embed_query_batch`), `embed`, the parity check and every CPU run stay on the batch-1 graphs. Measured by the issue on the RTX 4070 Laptop (E43, length 128): batch 1 100 docs/s, batch 4 166, batch 8 176, batch 16 64.
- New `EmbedderConfig::with_ingest_batch(n)` (the cap) and public field `ingest_batch` (default `DEFAULT_INGEST_BATCH` = 8; **Breaking (0.x minor)** for struct literals). Only 4 and 8 are accepted: anything else, 16 included, fails `load` with the new typed `EmbedError::UnsupportedIngestBatch { requested, reason }` (for 16 the reason names the 8 GB VRAM regression), and so does an ingest entry of another batch in a manifest. **Breaking (0.x minor):** `EmbedError` has the new variant.
- **int8 token table.** `token_table` `{"format": "int8-rowwise", "table": "embed_tokens.i8.npy", "scales": "embed_scales.f16.npy"}` loads the int8 `[262144, 512]` table and the fp16 `[262144, 1]` scales (a flat `[262144]` file is accepted too) and dequantises per looked-up row, `fp16 = i8 * scale`; the host table is 134 MB instead of 268 MB. Dtype, shape and byte length of both files are validated and a mismatch refuses the load with `InvalidConfig`. A missing `token_table`, or `{"format": "fp16", "table": "embed_tokens.f16.npy"}`, is the fp16 path.
- Pre-warm also builds the ingest graphs when the engine uses them, and every query length of the manifest.
- No CUDA or TensorRT path (decided: a 0 to 10 % gain for about 1 GB more download).
- Tests (unit, no model): the manifest and its strict refusals, template expansion, the batch rule per remaining count, the bucket choice, padding rows masked and dropped, input order across chunks, a hand-computed int8 dequantisation, shape and dtype refusals, the 4/8/16 cap. Written, unverified: nothing was compiled or run for this change.
## 0.5.0
- **Platform defaults.** Windows: E40 on DirectML with graph pre-warm and an opt-in parity verdict cache (load about 2 s warm against 42-51 s cold). macOS: the Core ML fp16 package with per-length compute units and pre-warm. Both are chosen from the model directory; the details are in the bullets below.
- **Pre-warm (E40 and Core ML).** New `EmbedderConfig::with_prewarm(bool)` (public field `prewarm: Option<bool>`; the default is on for Core ML and off for E40, because on Windows E40 pre-warm added 6-18 s to a 2 s cached load while a lazily built graph costs about 1-2 s on first use; **Breaking (0.x minor)** for struct literals). After admission every graph (E40) or function (Core ML) is built or loaded and has one dummy text run through it during `load`, and stays resident, so no query pays a graph build, compilation or shader build; `false` restores build-on-first-use. The time is in `LoadTimings::prewarm` (new `LoadTimings::prewarm_per_length`: per-function times, Core ML only). This supersedes the E40 bullets below that say graphs are released after admission. Ignored for CPU and Q4.
- **Core ML per-length compute units.** New public `CoreMlUnits`, `CoreMlComputeUnit` (`All`, `CpuAndNe`, `CpuAndGpu`, `CpuOnly`), `CoreMlUnitsParseError` and `NEURAL_ENGINE_MAX_TOKENS`. `CoreMlUnits::per_length_default()` is the default map, from the Mac matrix of RightKit issue #81: `cpu_and_ne` for the 32, 64 and 128 token functions (p50 about 2.3-3.0 ms) and `cpu_and_gpu` for 256 (about 5.1 ms) and 512 (about 9.8-18.7 ms). Each function loads on its own `MLModelConfiguration.computeUnits`; one `MLModel` per function and unit is kept resident. New `EmbedderConfig::with_coreml_units(impl Into<CoreMlUnits>)` (public field `coreml_units: Option<CoreMlUnits>`, **Breaking (0.x minor)** for struct literals) forces one unit for every length (`CoreMlComputeUnit::All` / `CpuAndNe` / `CpuAndGpu` / `CpuOnly`) or a custom map; `CoreMlUnits` parses `default`, a bare unit name, or `256=cpu_only,32=all` pairs over the default. New `Embedder::coreml_units()`. A map that is `cpu_only` everywhere is served as `"cpu"` with no probe, parity check or pre-warm. `Cpu` is unchanged. The first `cpu_and_gpu` call per length was measured at up to 51 s cold, which pre-warm moves into `load`. The compiled `.mlmodelc` (one per package-hash and OS key, in the caller's cache dir) is independent of the units, so changing the map never recompiles.
- Runtime budget: `Cargo.toml` now declares resources for `src/e40.rs` and `src/coreml.rs` (the `COMPILE` static is a data-free lock, with a `static_note`). `tests/runtime_budget_declaration.rs` pins the shape.
- **Load timings and parity verdict cache (E40).** New `Embedder::load_timings() -> Option<&LoadTimings>`: tokenizer load, token-table read, probe, every graph build (`SessionCreate`, per provider and length), the parity check (total, CPU reference, accelerator) and `total`. New opt-in `EmbedderConfig::with_parity_cache_dir(dir)` (public field `parity_cache_dir`, **Breaking (0.x minor)** for struct literals): the parity verdict is stored as one JSON file per model directory, keyed on the bundle files' names, sizes and mtimes, the ONNX Runtime library (path, size, mtime, sibling `DirectML.dll`, ORT build string) and the provider name (ORT exposes no cheap adapter identity; a stored verdict is trusted for 7 days because the driver is not in the key). A stored pass builds no CPU session; a stored failure still serves CPU; a missing, corrupt or non-matching file re-runs the check and rewrites it. No default location. New feature `bench-internals` (not default, not stable API): `Embedder::bench_query_tokens`, `bench_embed_query_split`. Written, unverified: not compiled or run.
- **E40 backend (Windows DirectML, CPU elsewhere).** A model directory holding `embed_tokens.f16.npy` and `model_32.onnx` (the layout of `scripts/embed-export/eg2-directml`) is loaded as the EmbeddingGemma 2 E40 bundle, chosen from the directory only. Five fixed-length fp16 graphs (32, 64, 128, 256, 512), batch 1; each text runs through the smallest graph that holds it, with the fp16 embedding row looked up on the host from the `.npy` table (unscaled) and an int64 mask; the graph returns the normalised 768-d vector (fp32 tail). Graphs are built on first use. Texts are truncated to 512 tokens. Sentinel scrubbing applies.
- Admission: `Auto` on Windows probes, then runs the existing vector parity check against the same graphs on the CPU provider. For E40 the check uses five prompts (about 20, 50, 90, 190 and 500+ tokens) so every graph is exercised; the graphs are released after admission and rebuilt on first use. `provider_used()` is `"directml"` or `"cpu"`; a graph that cannot be built on the requested provider is an error, never a silent CPU session. Explicit `DirectMl` fails with `AcceleratorParity` or a runtime error; `CoreMl` is refused for an E40 bundle.
- `Auto` not admitted: serves the E40 graphs on CPU, or, with the new `EmbedderConfig::with_fallback_model_dir(q4_dir)`, the Q4 bundle on CPU (its own `model_id()` and `pipeline_id()`, reason in `fallback_reason()`). **Breaking (0.x minor):** `EmbedderConfig` has a new public field `fallback_model_dir`.
- New `EMBEDDINGGEMMA2_E40_MODEL_ID` (`embeddinggemma2-e40-dml`) and `EMBEDDINGGEMMA2_COREML_MODEL_ID` (`embeddinggemma2-coreml`, now served by the Core ML backend below). `pipeline_id()` of an E40 bundle hashes the eight bundle files and uses its own model id, so it differs from the Q4 ids and consumers reindex.
- `rightkit-onnx` is now used with its `half` feature.
- **Core ML backend (macOS 15+).** A model directory holding `EmbeddingGemma2Text.mlpackage` and `embeddings.bf16` (the `embeddinggemma2-coreml` store entry: `FluidInference/embeddinggemma-2-coreml` at 3f978ddf) is loaded as the EmbeddingGemma 2 Core ML bundle, chosen from the directory only. ONNX Runtime cannot load an `.mlpackage`, so `src/coreml.rs` talks to Core ML directly through `objc2-core-ml` (macOS-only dependencies `objc2`, `objc2-core-ml`, `objc2-foundation`, same versions and features as `rightkit-asr`'s bridge, pulled by `inference` on macOS only). The package is a multi-function ML Program; functions `embed_32`, `embed_64`, `embed_128`, `embed_256`, `embed_512` (batch 1) are used (`embed_48` and `pack_256` are not). Each text runs through the smallest function that holds it (first 512 tokens kept). Inputs: `inputs_embeds` float16 `[1, L, 512]` (row of `embeddings.bf16`, bf16 widened by a 16-bit shift, times `embed_scale` = sqrt(512) from `config.json`, rounded to float16; pad rows scaled the same) and `attention_mask` float16 `[1, L]` (1 on tokens, 0 on right padding). Output `embedding` float32 `[1, 768]`, pooled, projected and L2-normalised in the graph. No ONNX Runtime is initialised for this bundle, so `EmbedderConfig::runtime_path` need not exist.
- Core ML compile and load: the package is compiled once with `MLModel.compileModelAtURL` into `<model_dir>/.coreml-compiled/embeddinggemma2-text-<key>.mlmodelc` (override: `EmbedderConfig::with_coreml_cache_dir`), where `<key>` hashes the three package files and the macOS version; an existing compile is reused and the install is atomic. Each function is loaded on first use with `MLModelConfiguration.functionName`; macOS older than 15 is refused with a clear `EmbedError::Runtime`. Core ML errors surface as `EmbedError::Runtime` with a `Core ML:` prefix. The first load compiles the Neural Engine program for all functions (the card says several minutes, once; Core ML caches it), so the first `Auto` load is slow.
- Core ML providers: `Auto` = all compute units (`provider_used()` `"coreml"`) after a probe and the existing vector parity check against the same package on `cpuOnly` (five prompts, every function exercised; functions are released after admission and loaded again on first use); explicit `CoreMl` is the same with no fallback (`AcceleratorParity` on a miss); `Cpu` = the package on `cpuOnly` (`"cpu"`); `DirectMl` is refused. With `EmbedderConfig::with_fallback_model_dir(q4_dir)`, `Auto` serves the Q4 bundle on CPU when the package cannot load (macOS older than 15, compile error), the probe fails or parity fails; without it a failed probe or parity check serves the package on `cpuOnly` and a load failure is an error. `pipeline_id()` hashes six files (the three package files, `embeddings.bf16`, `tokenizer.json`, `config.json`) with the Core ML model id. A Core ML bundle on a non-macOS host is refused with `InvalidConfig`.
- New: `EmbedderConfig::with_coreml_cache_dir` and public field `coreml_cache_dir` (**Breaking (0.x minor)** for struct literals, like `fallback_model_dir`).
- Tests: unit tests (no model, any platform) for the unit map, its override parsing and the budget declaration; unit tests (macOS) for bucket choice, bf16 widening and scaling, input padding, cache key, config checks and strided output reads. `tests/coreml_real_model_e2e.rs` (ignored by default; `RIGHTKIT_EMBED_GEMMA2_COREML_DIR`) checks admission, parity >= 0.995 against `cpuOnly` for every function length, ranking and explicit `CoreMl`.
- Written, unverified: nothing was compiled or run for this change; the per-length units, pre-warm and `LoadTimings` for Core ML were written on Windows and compile only on macOS. In particular the fp16 E40 graphs on the ONNX Runtime CPU provider (the parity reference) are unmeasured through this crate. The Core ML backend (objc2 calls, `setFunctionName`, compile and cache, parity of all compute units against `cpuOnly`) has never been built or run; the 12 Oct Mac measurement used Python coremltools, not this code.
## 0.4.0
- Republished on the renamed dependencies: `rightkit-onnx` (was `rightkit-ort`). The 0.3.2 on crates.io was packaged before the renames and still named the old crates.
- **Breaking (0.x minor):** `ExecutionProvider` converts from `rightkit_onnx::ExecutionProvider`.
- **An accelerator is used only if it reproduces the CPU provider's vectors at load.** DirectML
(ONNX Runtime 1.24.4) returns wrong EmbeddingGemma 2 Q4 vectors (cosine to CPU 0.58-0.72) that are
finite, non-zero and faster than CPU, which the probe and any speed check accept. Whenever the
session was built on anything other than CPU, `Embedder::load` now builds a CPU session for the same
model, embeds a short query, a medium document and a long document (each alone, then as one batch)
through both, and requires cosine at least `ACCELERATOR_PARITY_MIN_COSINE` (0.995) on every pair at
native dimension, with finite non-zero output. Under `Auto` a failed check falls back to CPU with
`fallback_reason` like `directml -> cpu: vector parity failed: cosine 0.579 at ~20 tokens, floor
0.995`. Under explicit `DirectMl` / `CoreMl` it fails the load with the new
`EmbedError::AcceleratorParity { provider, lowest_cosine, floor, detail }`. The CPU session is
dropped when the check passes; load costs one extra CPU session build and four ONNX calls per
session. `Cpu` is unchanged. The check cannot be turned off and the probe is unchanged.
- New public items: `ACCELERATOR_PARITY_MIN_COSINE`, `AcceleratorParity`,
`Embedder::accelerator_parity()`, `EmbedError::AcceleratorParity`. Adding an `EmbedError` variant
breaks exhaustive matches on it. `pipeline_id()` is unchanged by this (providers are not inputs).
- Consequence: `Auto` on Windows currently resolves to CPU for EmbeddingGemma 2 Q4 on the hardware
measured, and an explicit `DirectMl` errors. CoreML is unmeasured.
- Written, unverified: nothing was compiled or run for this change.
## 0.3.2
- Internal: model-digest and pipeline-fingerprint SHA-256 now go through `rightkit_fs::digest`
(`digest` feature only); the direct `sha2` 0.10 dependency is gone. Fingerprints are unchanged.
- `VectorStore::open` / `in_memory` open the connection through `rightkit-db`'s bootstrap
(`DbOptions::open_writer`, `open_in_memory`): WAL with verified fallback, 5 s busy timeout,
owner-only database file. Bootstrap failures that are not SQLite errors surface as
`VectorError::Sqlite` with `SQLITE_CANTOPEN` and the original message. `from_connection` is
unchanged for callers that bring their own connection.