Skip to main content

hive_leaf_format

Function hive_leaf_format 

Source
pub fn hive_leaf_format(dir: &Path) -> DirectoryFormat
Expand description

What the files under a hive root’s key=value partitions actually are.

A hive root holds no data itself, so directory_format can only say Deeper about one. This follows a single spine down — the same one path through the tree a hive scan reads its schema from — and reports what it finds at the bottom.

One spine, and the first partition at each level, so a dataset of ten thousand partitions costs what one of two costs. That makes it a sample: a tree whose partitions disagree is reported as whatever the first one holds. The alternative is walking the dataset to answer a question asked before it is opened.