pub fn embed_passages_parallel_shared(
_models_dir: &Path,
texts: Arc<[String]>,
parallelism: usize,
_local_batch_size: usize,
backends: BackendChoice,
) -> Result<Vec<Vec<f32>>, AppError>Expand description
Embeds many passages with EmbeddingBackendChoice awareness (GAP-SG-147).
THIS IS THE ONLY MULTI-PASSAGE ENTRY POINT. v1.0.93 (GAP-OR-INGEST): when
the resolved chain starts with OpenRouter and the client is initialised,
it uses the HTTP batch API (embed_batch) — no LLM slot consumed, ~200ms
per batch.
§Why the corpus arrives as an Arc<[String]>
A BORROWED slice cannot be handed to the 'static fan-out tasks, so an
entry point taking &[String] has to clone the entire corpus on every
call: a 36k-passage backfill copies every string before a single request
leaves the process. That borrowed-slice shim existed until v1.2.8 and was
removed; this signature is the reason it was never needed.
Taking ownership through an Arc<[String]> lets the OpenRouter fan-out
hand each task a refcount bump plus an index range instead of a cloned
Vec<String> per chunk. Arc::from(vec) MOVES the string buffers into the
Arc allocation — only the 24-byte headers are memcpy’d, never the heap
data — so the same 36k-text backfill copies nothing. Callers that hold a
Vec<String> pay one Arc::from(vec) and are done.
Chunk boundaries and ordering are unchanged: chunk i still covers
[i * chunk, min((i + 1) * chunk, len)) and reassemble_ordered still
sorts on that same index.
§Why local_batch_size reaches only ONE branch
The name is deliberate: this value governs the LOCAL (subprocess) branch and
is IGNORED under OpenRouter, which sizes its requests from XDG
embedding.batch_size through fan_out_chunk. That is not an oversight,
and “fixing” it would be a regression.
adaptive_batch_for_dim, which produces the value callers pass
here, was calibrated against SUBPROCESS backends. Its failure mode is an LLM
completing a prompt and truncating the JSON reply: at dim 384 with a fixed
batch of 8, claude returned 3 of 8 items and codex timed out at 300s.
Shrinking the batch as dimensionality grows is what keeps that from
happening.
The REST path cannot fail that way. OpenRouter exposes a native batch embedding API whose response is structured API JSON, not a model completion, so there is no token budget to truncate.
The cost of unifying them is concrete: adaptive_batch_for_dim(8, 1024)
resolves to 1 at this project’s active dimensionality. Letting the
dim-adaptive value win on the REST path would collapse every request to a
single text and destroy the 32x batching win of GAP-SG-141.
openrouter_branch_ignores_local_batch_size in this module’s tests fails if
the OpenRouter branch ever starts reading this parameter.