Skip to main content

embed_passages_parallel_shared

Function embed_passages_parallel_shared 

Source
pub fn embed_passages_parallel_shared(
    _models_dir: &Path,
    texts: Arc<[String]>,
    parallelism: usize,
    _local_batch_size: usize,
    backends: BackendChoice,
) -> Result<Vec<Vec<f32>>, AppError>
Expand description

Embeds many passages with EmbeddingBackendChoice awareness (GAP-SG-147).

THIS IS THE ONLY MULTI-PASSAGE ENTRY POINT. v1.0.93 (GAP-OR-INGEST): when the resolved chain starts with OpenRouter and the client is initialised, it uses the HTTP batch API (embed_batch) — no LLM slot consumed, ~200ms per batch.

§Why the corpus arrives as an Arc<[String]>

A BORROWED slice cannot be handed to the 'static fan-out tasks, so an entry point taking &[String] has to clone the entire corpus on every call: a 36k-passage backfill copies every string before a single request leaves the process. That borrowed-slice shim existed until v1.2.8 and was removed; this signature is the reason it was never needed.

Taking ownership through an Arc<[String]> lets the OpenRouter fan-out hand each task a refcount bump plus an index range instead of a cloned Vec<String> per chunk. Arc::from(vec) MOVES the string buffers into the Arc allocation — only the 24-byte headers are memcpy’d, never the heap data — so the same 36k-text backfill copies nothing. Callers that hold a Vec<String> pay one Arc::from(vec) and are done.

Chunk boundaries and ordering are unchanged: chunk i still covers [i * chunk, min((i + 1) * chunk, len)) and reassemble_ordered still sorts on that same index.

§Why local_batch_size reaches only ONE branch

The name is deliberate: this value governs the LOCAL (subprocess) branch and is IGNORED under OpenRouter, which sizes its requests from XDG embedding.batch_size through fan_out_chunk. That is not an oversight, and “fixing” it would be a regression.

adaptive_batch_for_dim, which produces the value callers pass here, was calibrated against SUBPROCESS backends. Its failure mode is an LLM completing a prompt and truncating the JSON reply: at dim 384 with a fixed batch of 8, claude returned 3 of 8 items and codex timed out at 300s. Shrinking the batch as dimensionality grows is what keeps that from happening.

The REST path cannot fail that way. OpenRouter exposes a native batch embedding API whose response is structured API JSON, not a model completion, so there is no token budget to truncate.

The cost of unifying them is concrete: adaptive_batch_for_dim(8, 1024) resolves to 1 at this project’s active dimensionality. Letting the dim-adaptive value win on the REST path would collapse every request to a single text and destroy the 32x batching win of GAP-SG-141.

openrouter_branch_ignores_local_batch_size in this module’s tests fails if the OpenRouter branch ever starts reading this parameter.