Trait Chunk 

pub trait Chunk: Sized + Clone {
    type Time: Lattice + Timestamp;

    const TARGET: usize;

    // Required methods
    fn len(&self) -> usize;
    fn merge(
        in1: &mut VecDeque<Self>,
        in2: &mut VecDeque<Self>,
        out: &mut VecDeque<Self>,
    );
    fn extract(
        input: &mut VecDeque<Self>,
        frontier: AntichainRef<'_, Self::Time>,
        residual: &mut Antichain<Self::Time>,
        keep: &mut VecDeque<Self>,
        ship: &mut VecDeque<Self>,
    );
    fn advance(
        input: &mut VecDeque<Self>,
        frontier: AntichainRef<'_, Self::Time>,
        done: bool,
        out: &mut VecDeque<Self>,
    );
    fn settle(input: &mut VecDeque<Self>, done: bool, out: &mut VecDeque<Self>);
}

A non-empty, bounded, consolidated, sorted sequence of (data, time, diff).

An implementor gains access to types and trait implementations that provide batch formation and trace maintenance with no additional effort.

The necessary implementations are either “data” or “metadata” operations. The “data” operations transform lists of chunks, are expected to do roughly “one chunk’s worth” of work at a time; they can afford to compress and page. The “metadata” operations provide chunk information, and should be lightweight.

The trait has no opinion about keys, vals, or diffs — only time, which trace maintenance needs. Reading a chunk’s contents is a separate, optional capability: see NavigableChunk.

Required Associated Constants

const TARGET: usize

The intended maximum chunk size.

Required Associated Types

type Time: Lattice + Timestamp

The timestamp type of the chunk’s updates.

Key/val/diff opinions live on the optional NavigableChunk capability; the chunk itself only needs time, to bound its interval and participate in advancement and compaction.

Required Methods

fn len(&self) -> usize

The number of updates in the chunk.

fn merge( in1: &mut VecDeque<Self>, in2: &mut VecDeque<Self>, out: &mut VecDeque<Self>, )

Merge the fronts of two input deques through their shared horizon.

Both deques are non-empty (the caller guarantees it). The two queues are both the heads of lists of chunks, and the implementor should only merge through the least last (key, val, time) update, or risk emitting an unconsolidated output chunk.

When a chunk cannot be completely retired, perhaps it had the larger last update, it should be rewritten as a new chunk and pushed back to the front of the queue. The invocation is expected to consume at least one of its inputs, and the harness may continually re-invoke if this doesn’t happen.

A merge concludes when the harness sees that either input is now empty, at which point it appends the queue to the output without the method’s assistance.

fn extract( input: &mut VecDeque<Self>, frontier: AntichainRef<'_, Self::Time>, residual: &mut Antichain<Self::Time>, keep: &mut VecDeque<Self>, ship: &mut VecDeque<Self>, )

Partition input updates into keep (greater or equal frontier) or not (ship).

An implementation should yield with some frequency to allow the output to “settle”. The harness may guard against this, but it prefers to provide as much context as it can in order to allow broader chunk fusion where needed.

fn advance( input: &mut VecDeque<Self>, frontier: AntichainRef<'_, Self::Time>, done: bool, out: &mut VecDeque<Self>, )

Advance times by frontier producing consolidated chunks.

An output for (key, val) should generally not be produced until a later pair is observed, or done is set, to ensure the output chunks are consolidated. Incomplete work can be pushed back to the front of input.

On done a single (key, val) group may span the whole input; advancing and consolidating it should cost time linear in its size, not quadratic.

fn settle(input: &mut VecDeque<Self>, done: bool, out: &mut VecDeque<Self>)

Reshape input to a sequence that maintains the “grading” structural invariant.

Specifically, the chunks in output should have a maximum size of TARGET and each adjacent pair should have lengths that sum to more than TARGET. This is also a good moment to consider compression or paging out the contents. When done is set the input must be moved to the output.

This method may be called on already settled data, and should be efficient then.

Implementors that want the standard maximal packing can delegate to the pack helper, supplying their layout’s coalesce / split / commit closures.

Dyn Compatibility

This trait is not dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors

impl<K, V, T, R> Chunk for VecChunk<K, V, T, R>

where K: Ord + Clone + 'static, V: Ord + Clone + 'static, T: Lattice + Timestamp, R: Ord + Semigroup + 'static,

const TARGET: usize = TARGET

type Time = <<Vector<((K, V), T, R)> as Layout>::TimeContainer as BatchContainer>::Owned

impl<U: ColumnarUpdate> Chunk for ColChunk<U>

where U::Time: 'static,

const TARGET: usize = TARGET

type Time = <<ColumnarLayout<U> as Layout>::TimeContainer as BatchContainer>::Owned