Module vector 

What a vector record holds, which is the vector itself at full precision.

06 section 2.1 gives kind 3 to a vector and says nothing about what is inside it, and 10 section 2 says why there is anything inside it at all: the searchable form of a vector is a RaBitQ code in a posting, and the code is lossy, so the last step of a search measures the best few candidates against the real thing. Every other engine that quantises has to keep the raw vectors somewhere on purpose. Here a vector is a record like any other and the rerank is a read at an address the id already resolves to.

+---------+---------+---------+---------+----------------------+
|   dim   | element |  flags  | reserved|  dim * width bytes   |
|    4    |    1    |    1    |    2    |                      |
+---------+---------+---------+---------+----------------------+

Why the dimension is in the record

The collection knows its own dimension, it is in the catalogue, and a reader that has the catalogue could work the count out from the record length. Storing it anyway costs four bytes and buys two things. A record is checkable on its own, so yodb check and the independent reader can say that a vector record is malformed without loading the catalogue for the collection it belongs to, and a collection whose dimension was changed under it produces a record that disagrees with the catalogue rather than a vector that is silently reinterpreted at a different length.

Why there is an element byte when there is one element type

Everything writes and reads Element::F32 today. The byte is there because after the freeze at the end of M6 the only lever left is min_reader_version, and half precision storage is the change most likely to be wanted: it halves what the log holds for a vector collection, which is most of what a vector collection is. With the byte here that lands as a new element value which old readers refuse one record at a time. Without it, it would need a new record kind or a format version, and both of those refuse the whole file.

An unknown element value is Code::Corrupt, not a record to skip. Skipping is the right answer for an unknown kind, because a kind a reader has never heard of is a thing it was never meant to understand. An element it cannot read inside a kind it can is different: the caller asked for this vector, and quietly returning nothing would look like a vector that is not there.

What is not in here

The codes, the centroids and the postings. Those are the index, the index is derived from these records, and 10 section 2 keeps them resident or in the cold tier rather than in the log. When they do get written down it will be as collection chunks under a checkpoint, because the record kinds are fixed by 06 and there is no kind for an index.

The id, because that is the record’s key, and the tag a filtered scan reads, because that is derived from the document the vector belongs to.

Structs

VectorBody

A vector record’s value, borrowed.

Enums

Element

How one coordinate is stored.

Constants

MAX_DIM

The largest dimension this version will write or read.

VECTOR_HEADER_LEN

The fixed part at the front of a vector record’s value.

Functions

vector_len

How long the value of a vector record with dim coordinates is.