Module chooser 

How the encoder decides which candidate to keep.

The encoders in crate::string and crate::integer are the work. This is the search. They are separate things and until now they were the same thing, because encode both offered every candidate and encoded every candidate it offered, and there was no way to have one without the other.

Why it is worth separating

cargo xtask encode over a million rows of ClickBench hits says where the encoder’s seconds go. String FRONT is 32.2 percent of them and is kept once in 142 chunks. String FSST is 11.2 percent and is kept twice in 223. Integer DELTA is 10.8 percent and is kept never in 607. String PLAIN is 8.7 percent and is kept four times in 223. Those four are 62.9 percent of the encoder’s time and they were kept seven times out of 1,195 offers.

That is not a bug in any encoder. It is what an exhaustive search costs, and the search is worth something: the shapes it arrives at are five to one on hits and nobody wrote them down in advance. The question is how much of the search is needed, which is a question about the data and therefore a question to measure rather than argue about. F2 asks for exactly this, as “the encoder chooser as a seam, with exhaustive and sampled implementations”, with the ablation being how much size the sampled one gives up.

What a chooser sees and what it does not

A chooser is asked once per chunk per level of the cascade, never once per value. It is handed the values and the candidates that apply and it returns the ones worth encoding in full. It cannot invent a candidate that does not apply, so nothing it does can produce a chunk that will not decode, and the worst a bad chooser can do is pick a bigger encoding than another one would have. That is the property that makes this safe to swap.

Not a rudb-seam seam yet, and why

SeamId::StorageEncoder exists and says “how a block of values is encoded on the way to disk”, and this is what belongs behind it. It cannot be registered here: rudb-seam is rank 2 and so is this crate, so the Strategy supertrait every seam trait needs is not visible from here. The registry goes in rudb-storage at rank 5, next to the write path, and there is no write path yet. Until there is, this is a plain trait with two implementations and an ablation, which is the part that can be measured today.

Structs

Exhaustive

Encode every candidate that applies and keep the smallest.

Sampled

Encode every candidate on a sample, then encode only the winner on the whole chunk.

Constants

EXHAUSTIVE

The one of these that does not have to be constructed, since it holds nothing.

Traits

Chooser

Which of the candidates that apply are worth encoding in full.