Struct Reader

Source

pub struct Reader { /* private fields */ }

Expand description

Reads committed native column pages without holding the table in memory.

Implementations§

Source §

impl Reader

Source

pub fn open(path: impl AsRef<Path>) -> Result<Self>

Opens the highest valid directory slot.

§Errors

If the file has no valid committed directory or a directory pointer is out of bounds.

Source

pub fn parts(&self) -> usize

How many parts the table has, which is how many chunks a scan of it reads.

Source

pub fn part_rows(&self, at: usize) -> usize

Rows in one part, or zero when the part number is past the table.

Source

pub fn table(&self) -> &Table

The committed table directory.

Source

pub fn top_frequencies( &self, column: usize, top: usize, ) -> Result<Option<Vec<(Value, u64)>>>

Exact leading frequencies when the stored synopsis proves a count-descending prefix.

The returned list can be longer than top. Keeping the stored tail lets a later TopN apply additional ordering keys without losing a value tied with the requested boundary.

§Errors

If the column is outside the schema or a stored value does not fit its declared type.

Source

pub fn frequency_occurrences( &self, column: usize, ) -> Result<Option<FrequencyOccurrences>>

Sparse rows belonging to the bounded numeric frequency candidate set.

The list is omitted when collecting it would exceed the fixed storage budget. A composite aggregate may accept a result over these rows only when its requested boundary is strictly greater than omitted_max.

§Errors

If the column is outside the schema.

Source

pub fn distinct_values(&self, column: usize) -> Result<Option<u64>>

How many distinct values one column holds, counting a null as no value.

A string column of this format is written against one dictionary that covers the whole table. A code is handed out the first time a value is seen and nothing ever removes one, so the number of codes is the number of distinct values exactly rather than an estimate. That makes COUNT(DISTINCT column) over a whole table a question the directory already knows the answer to, and the alternative is a hash table with a row per distinct value built from a pass over every row.

None for a column the file has no dictionary for, which is every column that is not a string, and None for a column with a null in it. A sketch would answer the first approximately and SQL asked for the exact number. The second is the placeholder: a null row is written as the code for the empty string, so a nullable column’s dictionary may hold an empty string that no row of it actually has, and nothing persisted today tells the two cases apart.

§Errors

If the column is outside the schema, or the dictionary page does not read.

Source

pub fn null_count(&self, column: usize) -> Result<u64>

How many rows of one column are null, added up over the stripes.

Every stripe records this exactly when it is written, because a null count is not a bound that is allowed to be wide the way a minimum and a maximum are: a filter that reads one too many is slow and a COUNT that reads one too many is wrong. Adding up a few hundred numbers already in memory is what makes COUNT(column) over a whole table free.

§Errors

If the column is outside the schema.

Source

pub fn text_extremes(&self, column: usize) -> Result<Option<(Value, Value)>>

The smallest and the largest value of one string column, from the order beside its values.

The dictionary holds exactly the values the column holds, so the first and the last of them in sorted order are the column’s minimum and maximum. Two reads of a rank block settle what otherwise walks a million rows.

None when the column is not a string, when the file was written before version 9 and so has no order, when the column has no values at all, or when it has a null in it, which is the placeholder again: the empty string a null is written as would sort ahead of every real value and be reported as the minimum.

§Errors

If the column is outside the schema, or a rank names a code the dictionary does not have.

Source

pub fn exact_extremes(&self, column: usize) -> Result<Option<(Bound, Bound)>>

The smallest and the largest value of one column, when every stripe wrote exact ends.

A stripe’s ends are allowed to be wider than the truth, because a bound that rules out a chunk that could not match is still correct when it rules out nothing. That is what makes them cheap to write for a bit packed or a dictionary column, and it is also what stops them answering a MIN. So each stripe says which of the two it wrote, and this answers only when all of them walked their rows.

None for a column with no ends, for an empty table, and for a column any stripe of which guessed. Nulls need no special case, because the ends skip them the same way MIN does.

One case is given up on that did not have to be. A stripe merges the ends of its sixty four parts, and a part with no ends at all erases the merged ones, because a part whose rows are not covered by the stripe’s ends is a stripe that would skip rows it should keep. A part of nothing but nulls has no rows to cover and so did not need to erase anything, but the merge cannot tell that part from a part whose layout it could not read. So a column with a chunk of nothing but nulls in the middle of it goes and reads the rows. That is slow and right, and the fix is a row count per part rather than anything here.

§Errors

If the column is outside the schema.

Source

pub fn exact_sum(&self, column: usize) -> Result<Option<(i128, u64)>>

The sum of one integer column and how many rows went into it, when every stripe wrote one.

The count beside the sum is the non-null rows, because that is what a SUM adds up and what an AVG divides by, and a caller that had to work it out from the row count and the null count would be doing the same walk twice.

None for anything that is not an integer column, for a file written by something that did not record it, and when adding the stripes together would overflow.

§Errors

If the column is outside the schema.

Source

pub fn read(&self, part: usize, columns: &[usize]) -> Result<Chunk>

Reads only the named columns from one part.

The whole stripe page each column lives in is read and kept, because a scan asks for the parts of a stripe one after another and this is what turns sixty four reads into one.

§Errors

If a part, column, page, or checksum is invalid.

Source

pub fn read_sparse(&self, part: usize, columns: &[usize]) -> Result<Chunk>

Reads named columns from one part without keeping the stripe page it came out of.

This is for sparse row fetches after a selective TopN or filter, which reach a few parts of a stripe rather than all of them. A caller that will read most of a stripe should use Self::read instead, because this reads and discards the page index every time.