Skip to main content

VariableColumn

Struct VariableColumn 

Source
pub struct VariableColumn { /* private fields */ }

Implementations§

Source§

impl VariableColumn

Source

pub fn new( data_path: PathBuf, offsets_path: PathBuf, ) -> Result<VariableColumn, Error>

Source

pub fn append_string(&mut self, value: &str) -> Result<(), Error>

Source

pub fn append_tagged(&mut self, tag: u8, value: &str) -> Result<(), Error>

Append a single tag byte followed by value’s bytes, as one row (#231).

Exactly equivalent to append_string(&format!("{tag_char}{value}")) for any tag that is a valid single-byte UTF-8 scalar, but without materializing the concatenation: the tag and the value are written from their own slices in one vectored write, so a tagged append costs no allocation and no copy of the value.

This is the write-side sibling of BufferedVariableColumn::read_str (#224) and it is schema-agnostic in exactly the same way — it knows nothing about what the tag means. Generated code uses it for the 1-byte presence tag on nullable string/json columns (0x00 = None, 0x01 = Some), but the column stores opaque bytes either way.

§Note on UTF-8

The stored row is only valid UTF-8 — and so only readable by read_string — when tag < 0x80. Tags at or above 0x80 produce a leading continuation/lead byte with nothing to complete it. The column itself is byte-oriented and will store them faithfully; it is the caller’s job to keep tags in the ASCII range if the row must round-trip as a String.

Source

pub fn read_string(&self, index: usize) -> Result<String, Error>

Source

pub fn gather_buffered( &self, indices: &[usize], ) -> Result<BufferedVariableColumn, Error>

Bulk-load the rows at indices into an in-memory BufferedVariableColumn whose read_string(slot) reads from memory (#168 column scan).

Reads only the spanned slice of the offsets index and maps only the spanned data region, so a scan over n variable rows pays one bounded read plus one mapping rather than 3n per-row read_exact_at syscalls (offset + length + data). indices are opaque physical row positions; reads no schema.

§Performance (#222)

This used to read the entire committed offsets index and the entire data region into owned buffers on every call, ignoring indices — dead versions included. That made a scan cost ~A x live_bytes, a slope in amplification rather than the fixed columns’ step, and it was measured as exactly that: at 2 000 live rows the narrow scan went 563 µs -> 5.8 ms across A = 1 -> 16, doubling with each doubling of A.

Both reads are now bounded to [min(indices), max(indices)]. Append order makes data offsets monotonic in row index, so that row span maps to one contiguous byte span. The data span is mapped rather than read when it is large enough to be worth it, which is what removes the slope: dead versions inside the span cost address space, not I/O, because only pages actually addressed by read_string are ever faulted in.

The residual cost is page granularity — a live row drags in its whole page, including any dead bytes sharing it — so the win depends on how clustered the live set is. Append-only helps here: an updated row is re-appended at the tail, so churn tends to concentrate live rows rather than scatter them.

§Sparse selections (#228)

Bounding to the span is the wrong trade when the selection is a handful of rows scattered across the column — the offsets read is then almost entirely rows nobody asked for. Below [SPARSE_OFFSETS_SPAN_FACTOR] density the offsets are read per index instead. This mirrors what FixedColumn::gather already does via GATHER_MMAP_MIN_ROWS; the data region still maps by span, which costs address space rather than I/O.

§Errors

InvalidInput if any index is >= self.len(); other errors come from the underlying reads.

Source

pub fn flush(&mut self) -> Result<(), Error>

Flush all pending writes to disk (fsync both data and offsets files).

After flush() returns Ok(()), all previous appends are guaranteed to survive a crash. Call at commit boundaries or before advancing the WAL checkpoint.

Source

pub fn sync_to_drive(&self) -> Result<(), Error>

Push both backing files to the drive cache without a device barrier (#153). Pair with one VariableColumn::barrier per checkpoint.

Source

pub fn barrier(&self) -> Result<(), Error>

Issue the single device-cache barrier for a checkpoint (#153). Barriers are device-wide, so one call on the data file covers the offsets file too.

Source

pub fn len(&self) -> usize

Source

pub fn is_empty(&self) -> bool

Source

pub fn sync_from_disk(&mut self) -> Result<(), Error>

Refresh the in-memory counters from the on-disk file lengths (#84).

Re-derives row_count from the offsets file (16 bytes/row) and current_data_offset from the data file, so a peer process’s appended rows (Tier 3 multi-process writer path) become readable through this handle. Paired with FixedColumn::sync_from_disk / Tombstones::sync_from_disk. A no-op for the single-writer path.

Source

pub fn truncate_to_rows(&mut self, rows: usize) -> Result<(), Error>

Truncate the column to hold exactly rows rows, discarding everything after that point.

The data file is cut at the byte immediately after the last byte of row rows - 1 (read from the offsets entry for that row). The offsets file is cut to rows * 16 bytes. Both in-memory counters are updated so the next append_string lands at exactly row rows.

§Errors

Returns Err(InvalidInput) if rows > self.len(). All other errors come from underlying I/O or set_len syscalls.

Source

pub fn reader(&self) -> Result<VariableColumnReader, Error>

Open a read-only, positionally-reading view over this column’s data + offsets files (#56 Direction B). The returned VariableColumnReader shares both files via independent (try_cloned) descriptors, so a single &mut self writer can keep appending while many &self readers read concurrently without a lock. See FixedColumnReader for the model.

Auto Trait Implementations§

Blanket Implementations§

Source§

impl<T> Any for T
where T: 'static + ?Sized,

Source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
Source§

impl<T> Borrow<T> for T
where T: ?Sized,

Source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
Source§

impl<T> From<T> for T

Source§

fn from(t: T) -> T

Returns the argument unchanged.

Source§

impl<T, U> Into<U> for T
where U: From<T>,

Source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

type Error = !

The type returned in the event of a conversion error.
Source§

fn try_from(value: U) -> Result<T, !>

Performs the conversion.
Source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.