Module vector 

Vectors, the collection of embeddings, and the search over it (10 and 15 section 2).

A vector collection is a name, a dimension and a metric. Something goes in under a key, the same thing comes back under that key, and a query vector gets the nearest keys to it. There is no index to create, no probe list to tune and no build step: the index is maintained as the collection is written to, in bounded pieces, which is what 10 section 5 is about and is the whole reason the index under here is partitions rather than a graph.

let db = yo::open(yo::MEMORY)?;
let v = db.vectors("passages", 3)?;

v.put("a", &[1.0, 0.0, 0.0])?;
v.put("b", &[0.0, 1.0, 0.0])?;

let hits = v.search(&[0.9, 0.1, 0.0], 1)?;
assert_eq!(hits[0].key, b"a".to_vec());

What a search costs, and why it is exact at the end

The searchable form of a vector is a RaBitQ code, which is a bit per dimension, so a collection of 768 dimensional embeddings is 96 bytes a vector to scan rather than 3072. The codes pick the candidates and then the full precision vectors settle the order, so a hit’s distance is the real distance and not an estimate, and the only thing quantisation can cost is a near miss that never made the shortlist. yo-vector’s recall tables are the measured version of that sentence.

The metric decides what is stored

Metric::L2 stores the vector it was given. Metric::Cosine stores the unit vector, because the index measures distance and on unit vectors the nearest by distance is the nearest by angle, which means cosine costs one normalisation on the way in rather than a different index. That is also what comes back out of Vectors::get, and it is the same answer Redis gives for a cosine vector set.

Metric::Ip and Metric::Hamming are refused rather than approximated. Inner product is not a distance, so ordering by it is not ordering by nearness and the partitions would be built around the wrong question, and Hamming wants binary vectors that this collection does not hold yet.

Where the vectors live

Beside the index, one flat run of floats per collection, which is the shape 06 gives them: a vector is a record like any other and the rerank is a read at an address the id already resolves to. This build holds that run in memory, exactly as every other collection here is held in memory, and the record kind it becomes on disk is already written down as yo_format::vector. Nothing on this page changes when the file arrives.

What is on this page and what is not

Only the handle. The collection itself is yo_vector::Collection, one crate down, because the vector commands on the wire need the same key table, the same slab of floats and the same metric handling that this does. Two doors into one store is Y23 and it is the reason INCR off a socket and Db::counter cannot drift apart either.

Structs

Match

One answer from a search.

Vectors

A collection of vectors, reached by Db::vectors.