Module vector 

A vector index over a path into a document (10 section 3).

An embedding is a field like any other. A document that carries one at $.embedding should be one write, and a nearest neighbour search over the collection should be one call that hands back documents, not ids to go and look up somewhere else. Every other engine makes this two stores joined on the id by the caller, and the join is where the two of them drift apart.

use yo_doc::{Builder, Docs, Key};

let mut docs = Docs::new();
docs.create_index("$.lang")?;
docs.create_vector_index("$.embedding", 3)?;

for (id, lang, v) in [
    ("a", "en", [1.0, 0.0, 0.0]),
    ("b", "fr", [0.9, 0.1, 0.0]),
    ("c", "en", [0.0, 0.0, 1.0]),
] {
    let mut b = Builder::new();
    b.begin_object()?;
    b.key(b"lang")?;
    b.text(lang)?;
    b.key(b"embedding")?;
    b.begin_array()?;
    for x in v {
        b.float(x)?;
    }
    b.end_array()?;
    b.end_object()?;
    let bytes = b.finish()?.to_vec();
    docs.put_bytes(id.as_bytes(), &bytes)?;
}

// Nearest overall, which is the French one.
let mut best = Vec::new();
docs.nearest("$.embedding", &[1.0, 0.05, 0.0], 2, |id, _, _| best.push(id.to_vec()))?;
assert_eq!(best[0], b"a".to_vec());
assert_eq!(best[1], b"b".to_vec());

// Nearest among the English ones, decided inside the scan and not after it.
let mut found = Vec::new();
let english = [("$.lang", Key::text("en"))];
docs.nearest_where("$.embedding", &[1.0, 0.05, 0.0], 2, &english, |id, _, _| {
    found.push(id.to_vec())
})?;
assert_eq!(found, [b"a".to_vec(), b"c".to_vec()]);

Why this is not a PathIndex

Every other index kind files a document under byte keys, and the lookup is equality or a range over those bytes. Nearness is neither. There is no key a query could ask for, the answer depends on all of the coordinates at once, and the structure that answers it is a partitioned quantised index rather than a table from key to posting list. So a vector index is a Collection, keyed by document id, held in its own list beside the path indexes rather than pretending to be one.

It is still the same Collection a vector set on the wire is, which is Y23 again: a document’s embedding and a VADD land in the same code, so a replace, a zero length vector and a dimension mismatch cannot be answered one way here and another way there.

The filter is the point

“The five nearest documents where the language is English” is the question people actually have, and answering it by searching for fifty and then throwing away the ones that are not English is a lottery. The more selective the filter the worse the lottery, and it fails quietly: the answers that come back are real, the ones that should have been there were never ranked.

So the filter runs inside the posting scan. Every document carries a 64 bit Signature over the keys its other indexes filed it under, which sits beside the code in the posting and costs one instruction to test on a word the scan has already loaded. nearest_where builds the same signature out of the values the query requires, and a document is worth ranking when its bits cover the query’s.

Two values can land on the same bit, so the signature can let a document through that does not really match. It can never reject one that does, which is the direction that matters, and the caller’s own predicate over the answers settles the rest.

Because the tag summarises the other indexes, it goes stale when the set of indexes changes. Declaring or dropping an index therefore rewrites every tag, which is one store per document per vector index with no requantising, rather than leaving a filter that used to work quietly answering nothing.

Structs

VectorIndex

One path holding an embedding, and the collection its vectors live in.