Skip to main content

is_nested

Function is_nested 

Source
pub fn is_nested(files: &[Vec<String>]) -> bool
Expand description

Whether every file’s columns are contained in the widest file’s.

The one shape schema evolution produces, and the one that reads cleanly as a union: a file written before a column existed has every column the widest file has, minus the ones added since. Nothing is scored and nothing is thresholded — a file either brings a column no other file has, or it does not.

This replaced a containment ratio against a 0.5 threshold. The ratio was measured and the threshold sat in a wide gap, but every counter-example found was a directory landing on the wrong side of a number: two tables joined on one key scored exactly 0.5, and a dataset grown from ten columns to fifty with one dropped along the way scored 0.196 — below the 0.200 of unrelated tables sharing a key. A dataset that drifted could not be told from tables that never agreed, by any statistic over column overlap, because the two produce the same overlaps. So the question changed instead of the number: not “how much do these agree” but “does any file bring something the others cannot account for”.

A directory that fails this is not refused. It is one keystroke further away — the row goes inside instead of opening, and the (all files) row inside it opens the union anyway. That is what makes a strict rule affordable here.

Fewer than two files is one table by definition, and so is a directory whose files all have no columns to disagree about.