Struct Config 

pub struct Config {
    pub threads: usize,
    pub coalesce_gap: u64,
    pub coalesce_span: u64,
    pub coalesce: bool,
}

How an Pool is sized and how hard it coalesces.

Fields

threads: usize

How many I/O threads. One means every batch is served in submission order by one thread, which is a useful thing for a test to be able to ask for.

coalesce_gap: u64

Two ranges no further apart than this are read as one, and the bytes in the gap are read and thrown away. Zero merges only ranges that touch.

coalesce_span: u64

A merged read never spans more than this, however small the gaps are. Without it a projection of two columns from opposite ends of a row group turns into a read of the row group.

coalesce: bool

Whether to merge at all.

Implementations

impl Config

pub fn local_disk() -> Self

The sizing for a local disk.

Every number here came out of cargo xtask io --cold on server3, which is what that task exists for. The warm table is the opposite of the cold one on almost every row, which is the reason the cold one is the one that decided this.

Threads equal to the core count, floored at two and capped at eight. Cold, on eight cores, a batch of scattered reads goes from 2.7 seconds through the loop to 392 milliseconds at eight threads, and sixteen threads is 413, inside the spread. Sequential is 204 at eight and 211 at sixteen, the same. Eight is where the device saturates and past it the extra threads are contending for the memory bandwidth the execution threads want.

Coalescing on, over gaps of sixteen kilobytes. The gap is the whole decision and it is a narrow one. Sixteen kilobytes takes a batch of scattered eight kilobyte pages from 413 milliseconds to 223, nearly twice as fast, and it does it while reading only 1.22 times the bytes asked for. Widening it to half a megabyte buys nothing on that pattern that is outside the spread, reads 7.03 times the bytes, and costs a column projection dearly: 64 kilobyte ranges 448 kilobytes apart go from 60 milliseconds unmerged to 115 merged, because the gaps between columns are real and reading them is work. Sixteen kilobytes is small enough to leave that pattern alone entirely, which is why it is the number.

The span cap means a sequential scan in one megabyte ranges merges nothing at all, which the table confirms: its read count does not move at any gap. That is the intended answer. A one megabyte read is already large enough that saving the syscall next to it is not measurable.

pub fn object_store() -> Self

The sizing for object storage.

An order of magnitude more threads, because the thing being hidden is a round trip rather than a device, and coalescing on with a generous gap, because a request that costs a hundred milliseconds however many bytes it asks for makes reading a gap and throwing it away obviously right.

Named rather than measured. There is no object store in this workspace yet and this is the default it will be measured against when there is, not a number that came out of a run.

pub fn with_threads(self, threads: usize) -> Self

This configuration with threads threads.

pub fn coalescing(self, gap: u64) -> Self

This configuration merging ranges no more than gap bytes apart.

pub fn not_coalescing(self) -> Self

This configuration issuing every range as its own read.

Trait Implementations

impl Clone for Config

fn clone(&self) -> Config

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for Config

impl Debug for Config

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for Config

fn default() -> Self

Returns the “default value” for a type.

impl Eq for Config

impl PartialEq for Config

fn eq(&self, other: &Config) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for Config