Struct Reader

Source

pub struct Reader { /* private fields */ }

Expand description

Reads committed native column pages without holding the table in memory.

Implementations§

Source §

impl Reader

Source

pub fn open(path: impl AsRef<Path>) -> Result<Self>

Opens the highest valid directory slot.

§Errors

If the file has no valid committed directory or a directory pointer is out of bounds.

Source

pub fn reads(&self) -> Reads

What this reader has read so far, and what opening it cost.

Public because the claim of spec/stats/04-in-memory.md section 4.2 is about this number and a claim nobody can check is a comment. A caller that wants to know whether opening a file touched the data asks here, and gets an answer that does not depend on what the page cache happened to hold.

Source

pub fn layout(&self) -> Layout

Where the file’s bytes went, from the directory alone.

No page is read, so this costs the same on a 45 GB table as on an empty one. See Layout for what is charged where and for why the three things that are not columns stay separate.

Source

pub fn parts(&self) -> usize

How many parts the table has, which is how many chunks a scan of it reads.

Source

pub fn stripe_parts(&self) -> Vec<Range<usize>>

The parts of each stripe, in table wide part numbers.

A scan that wants one worker to own the page it reads hands work out in these runs. The stripes are contiguous in part numbering and all but the last hold sixty four parts, but a stripe can be flushed early when rows arrive out of order, so the runs are read off the directory rather than worked out from a constant.

Source

pub fn keep_stripes(&self, stripes: usize)

Asks the page cache to keep stripes stripes of every column instead of the default.

This only ever raises the number. A scan that gives each worker a whole stripe has one page per column per worker open at once, and a cache smaller than that is worse than no cache at all: every worker’s page is evicted by the others before it has finished its stripe, so it reads a quarter of a megabyte for every part it takes out of it.

Source

pub fn part_rows(&self, at: usize) -> usize

Rows in one part, or zero when the part number is past the table.

Source

pub fn table(&self) -> &Table

The committed table directory.

Source

pub fn top_frequencies( &self, column: usize, top: usize, ) -> Result<Option<Vec<(Value, u64)>>>

Exact leading frequencies when the stored synopsis proves a count-descending prefix.

The returned list can be longer than top. Keeping the stored tail lets a later TopN apply additional ordering keys without losing a value tied with the requested boundary.

§Errors

If the column is outside the schema or a stored value does not fit its declared type.

Source

pub fn exact_frequencies( &self, column: usize, ) -> Result<Option<Vec<(Value, u64)>>>

Every value of one column with the number of rows holding it, when the synopsis is complete.

The heavy hitter pass keeps a bounded set of candidates and decrements them all when it runs out of room, so what it usually ends with is the leading values and a bound on everything it dropped. omitted_max of zero says that never happened: no candidate was ever decremented and the entries did not overflow the stored budget, so the list is every distinct value of the column with an exact count, and a null counts as a value of its own rather than being skipped.

That makes a whole class of question answerable without reading a row. How many rows hold a value, how many do not, and what a GROUP BY of that column with a count over it produces are all in here. It is only ever true of a column with few enough distinct values, which is the case worth having, because that is exactly the column a grouping or an equality filter would otherwise walk every row to answer.

None when the column has no synopsis, or has one that dropped anything.

§Errors

If the column is outside the schema or a stored value does not fit its declared type.

Source

pub fn frequency_occurrences( &self, column: usize, ) -> Result<Option<FrequencyOccurrences>>

Sparse rows belonging to the bounded numeric frequency candidate set.

The list is omitted when collecting it would exceed the fixed storage budget. A composite aggregate may accept a result over these rows only when its requested boundary is strictly greater than omitted_max.

§Errors

If the column is outside the schema.

Source

pub fn distinct_values(&self, column: usize) -> Result<Option<u64>>

How many distinct values one column holds, counting a null as no value.

A string column of this format is written against one dictionary that covers the whole table. A code is handed out the first time a value is seen and nothing ever removes one, so the number of codes is the number of distinct values exactly rather than an estimate. That makes COUNT(DISTINCT column) over a whole table a question the directory already knows the answer to, and the alternative is a hash table with a row per distinct value built from a pass over every row.

None for a column the file has no dictionary for, which is every column that is not a string, and None for a column with a null in it. A sketch would answer the first approximately and SQL asked for the exact number. The second is the placeholder: a null row is written as the code for the empty string, so a nullable column’s dictionary may hold an empty string that no row of it actually has, and nothing persisted today tells the two cases apart.

§Errors

If the column is outside the schema, or the dictionary page does not read.

Source

pub fn null_count(&self, column: usize) -> Result<u64>

How many rows of one column are null, added up over the stripes.

Every stripe records this exactly when it is written, because a null count is not a bound that is allowed to be wide the way a minimum and a maximum are: a filter that reads one too many is slow and a COUNT that reads one too many is wrong. Adding up a few hundred numbers already in memory is what makes COUNT(column) over a whole table free.

§Errors

If the column is outside the schema.

Source

pub fn text_extremes(&self, column: usize) -> Result<Option<(Value, Value)>>

The smallest and the largest value of one string column, from the order beside its values.

The dictionary holds exactly the values the column holds, so the first and the last of them in sorted order are the column’s minimum and maximum. Two reads of a rank block settle what otherwise walks a million rows.

None when the column is not a string, when the file was written before version 9 and so has no order, when the column has no values at all, or when it has a null in it, which is the placeholder again: the empty string a null is written as would sort ahead of every real value and be reported as the minimum.

§Errors

If the column is outside the schema, or a rank names a code the dictionary does not have.

Source

pub fn exact_extremes(&self, column: usize) -> Result<Option<(Bound, Bound)>>

The smallest and the largest value of one column, when every stripe wrote exact ends.

A stripe’s ends are allowed to be wider than the truth, because a bound that rules out a chunk that could not match is still correct when it rules out nothing. That is what makes them cheap to write for a bit packed or a dictionary column, and it is also what stops them answering a MIN. So each stripe says which of the two it wrote, and this answers only when all of them walked their rows.

None for a column with no ends, for an empty table, and for a column any stripe of which guessed. Nulls need no special case, because the ends skip them the same way MIN does.

One case is given up on that did not have to be. A stripe merges the ends of its sixty four parts, and a part with no ends at all erases the merged ones, because a part whose rows are not covered by the stripe’s ends is a stripe that would skip rows it should keep. A part of nothing but nulls has no rows to cover and so did not need to erase anything, but the merge cannot tell that part from a part whose layout it could not read. So a column with a chunk of nothing but nulls in the middle of it goes and reads the rows. That is slow and right, and the fix is a row count per part rather than anything here.

§Errors

If the column is outside the schema.

Source

pub fn exact_sum(&self, column: usize) -> Result<Option<(i128, u64)>>

The sum of one integer column and how many rows went into it, when every stripe wrote one.

The count beside the sum is the non-null rows, because that is what a SUM adds up and what an AVG divides by, and a caller that had to work it out from the row count and the null count would be doing the same walk twice.

None for anything that is not an integer column, for a file written by something that did not record it, and when adding the stripes together would overflow.

§Errors

If the column is outside the schema.

Source

pub fn read(&self, part: usize, columns: &[usize]) -> Result<Chunk>

Reads only the named columns from one part.

The whole stripe page each column lives in is read and kept, because a scan asks for the parts of a stripe one after another and this is what turns sixty four reads into one.

§Errors

If a part, column, page, or checksum is invalid.

Source

pub fn read_sparse(&self, part: usize, columns: &[usize]) -> Result<Chunk>

Reads named columns from one part without keeping the stripe page it came out of.

This is for sparse row fetches after a selective TopN or filter, which reach a few parts of a stripe rather than all of them. A caller that will read most of a stripe should use Self::read instead, because this reads and discards the page index every time.

§Errors

If a part, column, page, or checksum is invalid.

Source

pub fn skips_codes( &self, part: usize, column: usize, candidates: &[u32], ) -> Result<bool>

Whether an exact global-code membership index proves that the stripe holding a part cannot contain any of the sorted candidate codes.

§Errors

If the part, column, index page, checksum, or delta stream is invalid.

Source

pub fn skips(&self, part: usize, probes: &[Probe]) -> bool

Whether persisted statistics prove that a part cannot match the predicates.

Two of them. The bounds are per stripe, so every part of a stripe gets the same answer from those and a scan that skips one part that way skips all sixty four. The sieves are per part and answer equality, which is the test bounds are worst at: a column of identifiers has every stripe covering nearly the whole of its type, so the bounds keep every part of it and the sieve keeps the ones that really hold the value.

The bounds go first because they are already in memory and the sieves are a read.