Module image 

Writing a collection down, and reading it back without rebuilding it.

yo_format::image is the layout and this is the part that fills it in. The split is the same one the rest of the build keeps: the format crate knows where a byte goes and nothing else, and the crate that owns the structure knows what the byte means.

Why an index is written down at all

The records give back the vectors. They do not give back the shape. A collection of a million vectors is a few thousand partitions that arrived at their centroids through a long sequence of splits, merges and sweeps, and rebuilding that on open is a million quantisations and a lot of two means: minutes, on a machine that is meant to be answering. Every other index in this family has the same problem and most of them solve it by not solving it, which is what “the index warms up” means when a vendor says it.

So a checkpoint writes the index down and an open reads it back. Nothing is requantised on the way in, because requantising is the rebuild.

The two halves of a load

An image holds the codes and does not hold the vectors, and that is on purpose: the vectors are records of kind 3 already, at addresses the log resolves, and G8’s budget is 96 bytes of index for a 768 dimensional vector with the raw copy in the log. An image that carried them too would write every vector twice to save a walk.

So the caller brings them. Stored is that: something that can produce the vector a key was stored under, which for the engine is the log and for a test is a map. A key the store cannot produce is dropped rather than refused, which is the same answer Vectors gives for an id the log forgot: an index that heals is worth more than an index that is right about being unable to open.

What comes back out of the store has to be what went in, which for a cosine collection is the unit vector rather than whatever the client sent, because that is what the collection stored and what its codes were measured against. Collection::get returns the same thing, so a store built out of one collection reloads another exactly.

The order things are written in

Sections first, root last. A chain writes its chunks before its directory, so a directory that is readable has readable chunks, and this is the same rule one level up: a root that is readable points at sections that are all there. A crash between the two leaves chunks nobody points at, which is what compaction is for.

Structs

Restored

What came back from an image.

Traits

Stored

Where the full precision vectors come back from when an image is loaded.