Module multi 

Columns encoded together instead of one at a time.

spec/06-compression.md section 6.4 calls this multi-column compression and says it is the mechanism that has no equivalent in DuckDB. The idea is that columns are not independent, so encoding them independently throws away real redundancy. On ClickBench hits the obvious case is URL and Referer, which are both URLs drawn from the same universe, and the less obvious one is that URL, Referer, Title and the referer derived columns are all the same alphabet and could share one symbol table.

The three strategies

INDEPENDENT is each column encoded on its own by crate::string. It is the baseline the other two have to beat, and it is what a group falls back to when they do not.

SHARED_TABLE is one FSST symbol table trained on a sample of all the columns, with every column compressed against it. Storing one 255 symbol table rather than six saves almost nothing by itself. The effect that matters is that a table trained on the union has more evidence per symbol, so it compresses each column better than a table trained on that column alone would, and the columns that gain most are the small ones that never had enough bytes to train on.

SHARED_DICT is one dictionary holding the union of the values, with every column becoming an array of codes into it. A value that appears in three columns is stored once rather than three times. The dictionary is itself a string column, so it goes back through the string chooser and comes out FSST compressed, which means a shared dictionary is also a shared symbol table.

What decides

Here, measuring all three and keeping the smallest, for the same reason crate::string does it that way: M1 is measuring what the format can do rather than how fast a writer can decide. A write path cannot afford this. Section 6.4 says detection is by sampling pairs over a global sample at table level, recorded in the catalog as hints, with each row group checking only the hinted pairs. dictionary_groups is that pruning step, and it runs on sketches rather than on data, so the 5,460 pairs of a 105 column table cost a Jaccard estimate each rather than a pass over the column.

What is not here

Correlation encodings, which are the third form in section 6.4: column B stored as a function of column A, either as a per dictionary entry lookup for a functional dependency or as B minus f(A) for a numeric one. crate::sketch::dependence is the detection half of that and the encoding half is its own piece of work.

Global dictionaries, which are section 6.5 and are a dictionary across the whole table rather than across a group of columns in one chunk. The two compose, and the reason they are separate is that a global dictionary needs an incremental builder that can spill, which is open question five and is the thing most likely to make the idea impractical.

Enums

Strategy

How a column group is encoded. The discriminant is the tag byte and is part of the format.

Functions

decode_group

Decodes a group written by encode_group.

describe

The shape a group was encoded as, as a line of text.

dictionary_groups

Which columns should be considered for a shared dictionary, from one sketch per column.

encode_group

Encodes a group of string columns together, choosing whatever comes out smallest.

strategy_sizes

The size of every strategy that applies, which is the measurement section 6.4 is asking for.