Struct Chunk

Source

pub struct Chunk { /* private fields */ }

Expand description

A batch of columns of equal length.

Implementations§

Source §

impl Chunk

Source

pub fn new(columns: Vec<Vector>) -> Result<Self>

A chunk of columns, taking the row count from the first of them.

§Errors

If the columns are not all the same length, or if there are more rows than VECTOR_SIZE.

Source

pub fn with_rows(columns: Vec<Vector>, rows: usize) -> Result<Self>

A chunk of columns that is rows long, for the case where there are no columns to take the count from.

§Errors

If any column is not rows long, or if rows is more than VECTOR_SIZE.

Source

pub fn empty(types: &[LogicalType]) -> Self

A chunk of the given types with no rows in it.

What a scan of an empty table returns and what an operator returns when it is done. The types are kept, because a consumer asks a chunk what its columns are before it asks whether there are any.

Source

pub fn columns(&self) -> &[Vector]

The columns.

Source

pub fn column(&self, index: usize) -> Result<&Vector>

One column.

§Errors

If there is no column at index.

Source

pub fn into_columns(self) -> Vec<Vector>

The columns, given up.

Source

pub fn width(&self) -> usize

How many columns.

Source

pub fn len(&self) -> usize

How many rows.

Source

pub fn is_empty(&self) -> bool

Whether there are no rows.

Source

pub fn footprint(&self) -> usize

How many bytes of memory this chunk is holding.

What the memory limit charges for a chunk somebody kept. A chunk handed from one operator to the next and dropped is not charged at all, because charging it would count the same megabyte once per level of the tree, and the levels of the tree are not where a query runs out of memory.

Every size in this workspace counts the thing itself as well as what it owns, so a column’s own bytes are already in its own number and are not added again here.

Source

pub fn types(&self) -> Vec<LogicalType>

The type of each column.

Source

pub fn value_at(&self, row: usize, column: usize) -> Value

The value at a row and a column, or null if either is past the end.

The slow path, same as Vector::value_at. It is what a result set is read out with and what a test asserts on.

Source

pub fn row(&self, row: usize) -> impl Iterator<Item = Value> + '_

One row, left to right.

Source

pub fn select(self, selection: &Selection) -> Result<Self>

The rows a selection kept, without moving any of the values.

Every column becomes a dictionary vector whose codes are the selection, which is the form spec/07-execution.md section 7.1 asks a filter to produce rather than compacting. It takes the chunk by value because that is what makes it free: the payload is moved into the new vector rather than copied, so a filter that keeps one row in a thousand still costs the selection and nothing else.

§Errors

If the selection points past the end of the chunk.

Source

pub fn compact(self, selection: &Selection) -> Result<Self>

The rows a selection kept, copied, so that nothing downstream reads through an indirection.

The copying counterpart to Self::select, and the two exist because neither one is right twice. Which one to call is measured rather than argued, and the measurement says something other than what the argument does, so here is both.

The argument is that selecting pays nothing now and one redirection on every later read of every kept row, while compacting pays a copy now and nothing afterwards, so the deciding variable is selectivity: keep a few rows and select, keep most of them and compact. The measurement says the deciding variable is not selectivity at all, it is how many times the rows are read again afterwards, and selectivity barely moves the line. On server3, over a chunk of two integer columns, compacting loses to selecting at every selectivity from one percent to a hundred when there is one later pass over the kept rows, and beats it at every selectivity from one percent to a hundred when there are sixteen. With four later passes the two are within a few percent of each other everywhere. Put a varchar column in the chunk and compaction loses almost everywhere, because copying string bytes is most of what it costs and the dictionary it avoids is most of what it saves.

Which is why nothing in the streaming pipeline calls this yet. A filter today feeds an aggregate or a projection and that is one pass or two, and end to end on two million rows SELECT sum(a), sum(b), count(*) FROM t WHERE a > ? measures the same either way at one percent selectivity and fifty percent slower compacting at fifty percent selectivity. The operators that will want this are the ones that hold chunks rather than pass them on, the hash join build side and the sort, because a chunk that is kept alive as a selection keeps the whole chunk it was selected from alive with it, and that is a hundred to one on memory rather than a few percent on time.

Takes the chunk by value like Self::select does, even though the payload is copied rather than moved, because a caller that still wanted the original after compacting it would be holding both copies and should say so.

§Errors

If the selection points past the end of the chunk, or if a column has a type with no flat layout, which today means the nested types.

Source

pub fn project(self, positions: &[usize]) -> Result<Self>

The columns at the given positions, in that order.

A position may appear twice, which is what SELECT x, x FROM t is, and the second one costs a copy. Every other position is moved.

§Errors

If a position is past the end of the chunk.

Source

pub fn flatten(&self) -> Result<Self>

The same rows with every column in flat form.

Costs a copy per column that was not already flat. It is here for the result set at the top of a query, where the dictionary vectors a filter left behind would otherwise be handed to a caller who has to understand them.