Module vector 

The vector itself.

spec/07-execution.md section 7.1 calls this the widest interface in the system, says every operator depends on it, and says changing it after twenty operators exist is expensive. So it is written before the first operator rather than after the fifth.

A vector is a type, a length of at most VECTOR_SIZE, a physical form, a validity representation and some data. Four of the forms are the ones in spec/04-architecture.md section 4.3: flat, constant, sequence and dictionary. Run length, bit packed and string view come after them, one at a time with the kernels that read them rather than all at once ahead of anything that can use them.

Dictionary and run length are the pair worth understanding together, because they answer different questions about the same column. A dictionary says which distinct values there are, so it wins on low cardinality however the rows are ordered. Run length says where the values stop, so it wins on a clustered column however many distinct values it has. A column can want either one without wanting the other, and hits has columns of both kinds.

String view is the odd one out, because it is not about making a column smaller. It is about who owns the bytes: the views are the vector’s and the arena is shared, so cutting a chunk out of a page of strings moves sixteen bytes a row and copies none of the payload. Every other form here trades a little work per row for less memory, and that one trades nothing at all.

The nested forms are the odd ones out in a different direction. The forms above are all ways of writing a column of scalars down more cheaply, and a nested value is not a scalar at all, so Form::List and Form::Struct are each the only form their column has rather than one of several it could be in. A list is a child vector of every element plus a start and a length per row. A struct is one child per field with no entries at all, because a struct row holds one value per field rather than a run of them. Either way the children are ordinary vectors and can be in any of the forms above, which is where a nested column gets made smaller.

What is not here yet. Buffers are owned. Section 7.1 says a vector borrowed from a buffer managed page carries a pin, and there is no buffer manager until M2, so there is nothing to pin and pretending otherwise would be an interface built against an imaginary caller. ARRAY is not stored yet either, and it is a composition of what is here rather than a new shape: it is a list whose length is the type’s rather than the row’s, the way a MAP is a list whose child is a two field struct of keys and values. UNION is the one that is genuinely different, since it is one child per member plus a tag saying which member each row is in.

Structs

Coded

The codes of a compressed column and the table they are against.

Packed

The bits of a packed vector and what they mean, for a kernel that wants to stay in code space.

Vector

A type, a length, a validity representation and some data.

Enums

Data

The values of a flat vector, one Rust vector per physical type.

Form

Which physical form a vector is in.

Constants

FSST_PAYS_AT

How much smaller compressing has to be before it is worth a decompression on every read.

MAP_KEY

What the key field of a map’s child struct is called.

MAP_VALUE

What the value field of a map’s child struct is called. See MAP_KEY.

PACKED_WIDTH_MAX

The widest a packed code is allowed to be.

PACKING_PAYS_AT

How much smaller packing has to be before it is worth the shift and the mask on every read.

VECTOR_SIZE

How many values are in a full vector.

Type Aliases

MapParts

What Vector::map_parts hands back: one entry per row, then the keys and then the values.