Struct Reader 

pub struct Reader { /* private fields */ }

Reads committed native column pages without holding the table in memory.

Implementations

impl Reader

pub fn open(path: impl AsRef<Path>) -> Result<Self>

Opens a file that holds exactly one table.

Errors

If the file has no valid committed directory, a directory pointer is out of bounds, or the file holds more than one table, which is a file that has to be opened by name.

pub fn reads(&self) -> Reads

What this reader has read so far, and what opening it cost.

Public because the claim of spec/stats/04-in-memory.md section 4.2 is about this number and a claim nobody can check is a comment. A caller that wants to know whether opening a file touched the data asks here, and gets an answer that does not depend on what the page cache happened to hold.

pub fn layout(&self) -> Layout

Where the file’s bytes went, from the directory alone.

No page is read, so this costs the same on a 45 GB table as on an empty one. See Layout for what is charged where and for why the three things that are not columns stay separate.

pub fn stored(&self, column: usize) -> Result<Vec<StoredPart>>

What every part of one column is stored as, which is what pragma_storage_info reports.

Unlike Self::layout this reads the data, because the encoder’s choice is in the page and nowhere else. The directory says how many bytes a column took and says nothing about what shape they are in, and the shape is the question worth asking: the same rows in a different order come back bit packed on one file and plain on another, and that is the difference a clustered load makes to a scan.

One read per stripe rather than one per part. A part is a few kilobytes out of a page that is a quarter of a megabyte, so asking part by part would read the same page sixty four times. Nothing is put in the page cache, because a caller asking what a file looks like is not about to scan it and evicting the pages a real query wants would be a poor trade.

Errors

If the column is outside the schema, or a page, index section or checksum is invalid.

pub fn parts(&self) -> usize

How many parts the table has, which is how many chunks a scan of it reads.

pub fn stripe_parts(&self) -> Vec<Range<usize>>

The parts of each stripe, in table wide part numbers.

A scan that wants one worker to own the page it reads hands work out in these runs. The stripes are contiguous in part numbering and all but the last hold sixty four parts, but a stripe can be flushed early when rows arrive out of order, so the runs are read off the directory rather than worked out from a constant.

pub fn stripe_rows(&self, stripe: usize) -> usize

How many rows one stripe holds, in the numbering Self::stripe_parts hands back.

Off the directory, which is already in memory, rather than by the caller asking for each part in turn through the catalog. Nothing past the end holds any rows.

pub fn keep_stripes(&self, stripes: usize)

Asks the page cache to keep stripes stripes of every column instead of the default.

This only ever raises the number. A scan that gives each worker a whole stripe has one page per column per worker open at once, and a cache smaller than that is worse than no cache at all: every worker’s page is evicted by the others before it has finished its stripe, so it reads a quarter of a megabyte for every part it takes out of it.

pub fn part_rows(&self, at: usize) -> usize

Rows in one part, or zero when the part number is past the table.

pub fn table(&self) -> &Table

The committed table directory.

pub fn top_frequencies( &self, column: usize, top: usize, ) -> Result<Option<Vec<(Value, u64)>>>

Exact leading frequencies when the stored synopsis proves a count-descending prefix.

The returned list can be longer than top. Keeping the stored tail lets a later TopN apply additional ordering keys without losing a value tied with the requested boundary.

Errors

If the column is outside the schema or a stored value does not fit its declared type.

pub fn top_pair_frequencies( &self, first: usize, second: usize, top: usize, ) -> Result<Option<PairFrequencyCounts>>

Exact leading counts for a numeric key paired with a stable-dictionary string key.

The stored prefix is returned only when its requested boundary strictly beats the bound on every pair omitted at load time. The returned tail may be longer than top, as with Self::top_frequencies, so downstream ordering can settle ties without reading rows.

Errors

If either column is outside the schema or persisted pair metadata is inconsistent with the frequency synopsis or dictionary it names.

pub fn exact_frequencies( &self, column: usize, ) -> Result<Option<Vec<(Value, u64)>>>

Every value of one column with the number of rows holding it, when the synopsis is complete.

The heavy hitter pass keeps a bounded set of candidates and decrements them all when it runs out of room, so what it usually ends with is the leading values and a bound on everything it dropped. omitted_max of zero says that never happened: no candidate was ever decremented and the entries did not overflow the stored budget, so the list is every distinct value of the column with an exact count, and a null counts as a value of its own rather than being skipped.

That makes a whole class of question answerable without reading a row. How many rows hold a value, how many do not, and what a GROUP BY of that column with a count over it produces are all in here. It is only ever true of a column with few enough distinct values, which is the case worth having, because that is exactly the column a grouping or an equality filter would otherwise walk every row to answer.

None when the column has no synopsis, or has one that dropped anything.

Errors

If the column is outside the schema or a stored value does not fit its declared type.

pub fn frequency_prefix(&self, column: usize) -> Result<Option<FrequencyPrefix>>

Every value the synopsis lists with the number of rows holding it, and a bound on the rest.

The counts are exact whether or not the list is complete. The heavy hitter pass keeps a bounded candidate set and then recounts only the candidates that survived it, so a value that made it into the list carries the number of rows that really hold it rather than whatever the pass had left over. What the pass loses is values, not counts.

omitted_max is how many rows the most common value left out can hold, and zero says nothing was left out at all, which is what exact_frequencies asks for. Above zero the list is the leading values of the column and everything else is somewhere between no rows and that bound.

That prefix is worth reading on its own. A column with a value in half its rows and a long tail behind it has no complete synopsis and never will, and it is the column where dividing the rows by the distinct count is furthest from the truth.

None when the column has no synopsis.

Errors

If the column is outside the schema or a stored value does not fit its declared type.

pub fn frequency_occurrences( &self, column: usize, ) -> Result<Option<FrequencyOccurrences>>

Sparse rows belonging to the bounded numeric frequency candidate set.

The list is omitted when collecting it would exceed the fixed storage budget. A composite aggregate may accept a result over these rows only when its requested boundary is strictly greater than omitted_max.

Errors

If the column is outside the schema.

pub fn distinct_values(&self, column: usize) -> Result<Option<u64>>

How many distinct values one column holds, counting a null as no value.

A string column of this format is written against one dictionary that covers the whole table. A code is handed out the first time a value is seen and nothing ever removes one, so the number of codes is the number of distinct values exactly rather than an estimate. That makes COUNT(DISTINCT column) over a whole table a question the directory already knows the answer to, and the alternative is a hash table with a row per distinct value built from a pass over every row.

A null in the column used to make this None and no longer does. A null row is written as the code for the empty string, so a nullable column’s dictionary can hold an empty string that no row of it actually has, and the dictionary on its own does not say which case it is. The writer does know, because it counts the non-null rows that use each code on its way to the frequency summary, so it records how many codes any row holds and the directory carries that number. This reads it rather than the size of the dictionary, which also means the dictionary page is not opened to answer.

An integer column has no dictionary, and its count comes from the set the writer keeps on its numeric frequency pass instead, which is exact up to a cap. None for a column past that cap and for every column that is neither, where a sketch would answer approximately and SQL asked for the exact number.

Errors

If the column is outside the schema.

pub fn null_count(&self, column: usize) -> Result<u64>

How many rows of one column are null, added up over the stripes.

Every stripe records this exactly when it is written, because a null count is not a bound that is allowed to be wide the way a minimum and a maximum are: a filter that reads one too many is slow and a COUNT that reads one too many is wrong. Adding up a few hundred numbers already in memory is what makes COUNT(column) over a whole table free.

Errors

If the column is outside the schema.

pub fn text_extremes(&self, column: usize) -> Result<Option<(Value, Value)>>

The smallest and the largest value of one string column, from the order beside its values.

The dictionary holds exactly the values the column holds, so the first and the last of them in sorted order are the column’s minimum and maximum. Two reads of a rank block settle what otherwise walks a million rows.

None when the column is not a string, when the file was written before version 9 and so has no order, when the column has no values at all, or when it has a null in it, which is the placeholder again: the empty string a null is written as would sort ahead of every real value and be reported as the minimum.

Errors

If the column is outside the schema, or a rank names a code the dictionary does not have.

pub fn exact_extremes(&self, column: usize) -> Result<Option<(Bound, Bound)>>

The smallest and the largest value of one column, when every stripe wrote exact ends.

A stripe’s ends are allowed to be wider than the truth, because a bound that rules out a chunk that could not match is still correct when it rules out nothing. That is what makes them cheap to write for a bit packed or a dictionary column, and it is also what stops them answering a MIN. So each stripe says which of the two it wrote, and this answers only when all of them walked their rows.

None for a column with no ends, for an empty table, and for a column any stripe of which guessed. Nulls need no special case, because the ends skip them the same way MIN does.

One case is given up on that did not have to be. A stripe merges the ends of its sixty four parts, and a part with no ends at all erases the merged ones, because a part whose rows are not covered by the stripe’s ends is a stripe that would skip rows it should keep. A part of nothing but nulls has no rows to cover and so did not need to erase anything, but the merge cannot tell that part from a part whose layout it could not read. So a column with a chunk of nothing but nulls in the middle of it goes and reads the rows. That is slow and right, and the fix is a row count per part rather than anything here.

Errors

If the column is outside the schema.

pub fn exact_sum(&self, column: usize) -> Result<Option<(i128, u64)>>

The sum of one integer column and how many rows went into it, when every stripe wrote one.

The count beside the sum is the non-null rows, because that is what a SUM adds up and what an AVG divides by, and a caller that had to work it out from the row count and the null count would be doing the same walk twice.

None for anything that is not an integer column, for a file written by something that did not record it, and when adding the stripes together would overflow.

Errors

If the column is outside the schema.

pub fn host_groups( &self, column: usize, minimum_count: u64, ) -> Result<Option<Vec<HostEntry>>>

Certified host groups over a string column, when the caller’s inclusive row-count bound excludes every host the synopsis omitted.

pub fn extents(&self, of: &Section) -> Result<Vec<Extent>>

Reads one section’s extent table and checks it against the entry that names it.

Errors

If the entry points outside the file, the table does not checksum, or it does not decode as a run of extents in element order.

pub fn extent(&self, of: &Extent) -> Result<Vec<u8>>

Reads and verifies one extent of a section.

This is what section 3.2’s second rule is for. A reduction that needs one extent of a two gigabyte forward link reads and checksums that extent and nothing else, which is the whole difference between a structure that works at SF100 and issue #745.

Errors

If the extent points outside the file, or its bytes do not checksum.

pub fn payload(&self, of: &Section) -> Result<Vec<u8>>

Reads a whole section’s payload, every extent of it, in order.

For a structure that is resident anyway, which a key map is. Anything large enough that the split matters should be walking Reader::extents and taking the one it needs.

Errors

If the extent table or any extent fails its check.

pub fn read(&self, part: usize, columns: &[usize]) -> Result<Chunk>

Reads only the named columns from one part.

The whole stripe page each column lives in is read and kept, because a scan asks for the parts of a stripe one after another and this is what turns sixty four reads into one.

Errors

If a part, column, page, or checksum is invalid.

pub fn read_sparse(&self, part: usize, columns: &[usize]) -> Result<Chunk>

Reads named columns from one part without keeping the stripe page it came out of.

This is for sparse row fetches after a selective TopN or filter, which reach a few parts of a stripe rather than all of them. A caller that will read most of a stripe should use Self::read instead, because this reads and discards the page index every time.

Errors

If a part, column, page, or checksum is invalid.

pub fn skips_codes( &self, part: usize, column: usize, candidates: &[u32], ) -> Result<bool>

Whether an exact global-code membership index proves that the stripe holding a part cannot contain any of the sorted candidate codes.

Errors

If the part, column, index page, checksum, or delta stream is invalid.

pub fn skips(&self, part: usize, probes: &[Probe]) -> bool

Whether persisted statistics prove that a part cannot match the predicates.

Three of them, asked cheapest first.

The stripe’s bounds are in memory already, so they are free, and they are also the coarsest: every part of a stripe gets the same answer and a scan that skips one part that way skips all sixty four. Then the part’s own bounds, which are a read of one page per column per stripe and are sixty four times finer. Then the sieves, which are per part and answer equality, the test bounds are worst at: a column of identifiers has every stripe and nearly every part covering the whole of its type, so bounds keep them all and the sieve keeps the ones that really hold the value.

The middle one is what an ordered comparison on a column the rows are not sorted by needs. On ClickBench 24 the stripe bounds leave eight stripes of sixteen alive, which is half the file, and the part bounds leave thirty parts of nine hundred and seventy four.

pub fn certain(&self, part: usize, probes: &[Probe]) -> bool

Whether persisted statistics prove that every row of a part matches the predicates.

Only the bounds. The sieves say nothing here, because a sieve that holds a value is a sieve that may be holding somebody else’s hash, so it can rule a part out and can never wave one through.

The stripe first and the part after it, the same two steps and in the same order as Self::skips. The stripe’s bounds are in memory already and its null count covers sixty four parts rather than one, so a stripe that answers is an answer for nothing, and the part’s own bounds are only read for the probes it could not settle. Both directions are safe: a stretch where everything passes contains no narrower stretch where something fails, and a stripe with no nulls has no nulls in any of its parts.

A string end a part recorded is cut down to its first few bytes, so a part’s stretch can be wider than its rows really are as well. That is the same safe direction for the same reason, and it is why this asks the two ends rather than anything exact says.

pub fn stripe_skips(&self, stripe: usize, probes: &[Probe]) -> bool

Whether the bounds of one stripe prove that none of its parts can match the predicates.

The cheap half of Self::skips, asked about a whole stripe at once. The bounds live in the directory and are already in memory, so this answers without touching the file, and that is the reason it is worth having on its own: a caller that wants to know roughly where the work is before it starts any workers can ask this about sixteen stripes for nothing, where asking Self::skips about nine hundred parts would read and decode a sieve page per stripe first.

It keeps stripes that Self::skips would rule out part by part, which is the right way for it to be wrong: the parts are still checked when they are read.

Trait Implementations

impl Clone for Reader

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Reader

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.