pub struct VariableColumn { /* private fields */ }Implementations§
Source§impl VariableColumn
impl VariableColumn
pub fn new( data_path: PathBuf, offsets_path: PathBuf, ) -> Result<VariableColumn, Error>
pub fn append_string(&mut self, value: &str) -> Result<(), Error>
Sourcepub fn append_tagged(&mut self, tag: u8, value: &str) -> Result<(), Error>
pub fn append_tagged(&mut self, tag: u8, value: &str) -> Result<(), Error>
Append a single tag byte followed by value’s bytes, as one row (#231).
Exactly equivalent to append_string(&format!("{tag_char}{value}")) for any
tag that is a valid single-byte UTF-8 scalar, but without materializing
the concatenation: the tag and the value are written from their own slices
in one vectored write, so a tagged append costs no allocation and no copy
of the value.
This is the write-side sibling of BufferedVariableColumn::read_str
(#224) and it is schema-agnostic in exactly the same way — it knows nothing
about what the tag means. Generated code uses it for the 1-byte presence
tag on nullable string/json columns (0x00 = None, 0x01 = Some), but
the column stores opaque bytes either way.
§Note on UTF-8
The stored row is only valid UTF-8 — and so only readable by
read_string — when tag < 0x80. Tags at or
above 0x80 produce a leading continuation/lead byte with nothing to
complete it. The column itself is byte-oriented and will store them
faithfully; it is the caller’s job to keep tags in the ASCII range if the
row must round-trip as a String.
pub fn read_string(&self, index: usize) -> Result<String, Error>
Sourcepub fn gather_buffered(
&self,
indices: &[usize],
) -> Result<BufferedVariableColumn, Error>
pub fn gather_buffered( &self, indices: &[usize], ) -> Result<BufferedVariableColumn, Error>
Bulk-load the rows at indices into an in-memory
BufferedVariableColumn whose read_string(slot) reads from memory
(#168 column scan).
Reads only the spanned slice of the offsets index and maps only the
spanned data region, so a scan over n variable rows pays one bounded
read plus one mapping rather than 3n per-row read_exact_at syscalls
(offset + length + data). indices are opaque physical row positions;
reads no schema.
§Performance (#222)
This used to read the entire committed offsets index and the entire
data region into owned buffers on every call, ignoring indices — dead
versions included. That made a scan cost ~A x live_bytes, a slope in
amplification rather than the fixed columns’ step, and it was measured as
exactly that: at 2 000 live rows the narrow scan went 563 µs -> 5.8 ms
across A = 1 -> 16, doubling with each doubling of A.
Both reads are now bounded to [min(indices), max(indices)]. Append order
makes data offsets monotonic in row index, so that row span maps to one
contiguous byte span. The data span is mapped rather than read when it
is large enough to be worth it, which is what removes the slope: dead
versions inside the span cost address space, not I/O, because only pages
actually addressed by read_string are ever faulted in.
The residual cost is page granularity — a live row drags in its whole page, including any dead bytes sharing it — so the win depends on how clustered the live set is. Append-only helps here: an updated row is re-appended at the tail, so churn tends to concentrate live rows rather than scatter them.
§Sparse selections (#228)
Bounding to the span is the wrong trade when the selection is a handful of
rows scattered across the column — the offsets read is then almost entirely
rows nobody asked for. Below [SPARSE_OFFSETS_SPAN_FACTOR] density the
offsets are read per index instead. This mirrors what
FixedColumn::gather already does via GATHER_MMAP_MIN_ROWS; the data
region still maps by span, which costs address space rather than I/O.
§Errors
InvalidInput if any index is >= self.len(); other errors come from the
underlying reads.
Sourcepub fn flush(&mut self) -> Result<(), Error>
pub fn flush(&mut self) -> Result<(), Error>
Flush all pending writes to disk (fsync both data and offsets files).
After flush() returns Ok(()), all previous appends are guaranteed to
survive a crash. Call at commit boundaries or before advancing the WAL checkpoint.
Sourcepub fn sync_to_drive(&self) -> Result<(), Error>
pub fn sync_to_drive(&self) -> Result<(), Error>
Push both backing files to the drive cache without a device barrier
(#153). Pair with one VariableColumn::barrier per checkpoint.
Sourcepub fn barrier(&self) -> Result<(), Error>
pub fn barrier(&self) -> Result<(), Error>
Issue the single device-cache barrier for a checkpoint (#153). Barriers are device-wide, so one call on the data file covers the offsets file too.
pub fn len(&self) -> usize
pub fn is_empty(&self) -> bool
Sourcepub fn sync_from_disk(&mut self) -> Result<(), Error>
pub fn sync_from_disk(&mut self) -> Result<(), Error>
Refresh the in-memory counters from the on-disk file lengths (#84).
Re-derives row_count from the offsets file (16 bytes/row) and
current_data_offset from the data file, so a peer process’s appended
rows (Tier 3 multi-process writer path) become readable through this
handle. Paired with FixedColumn::sync_from_disk /
Tombstones::sync_from_disk. A no-op for the single-writer path.
Sourcepub fn truncate_to_rows(&mut self, rows: usize) -> Result<(), Error>
pub fn truncate_to_rows(&mut self, rows: usize) -> Result<(), Error>
Truncate the column to hold exactly rows rows, discarding everything
after that point.
The data file is cut at the byte immediately after the last byte of row
rows - 1 (read from the offsets entry for that row). The offsets file
is cut to rows * 16 bytes. Both in-memory counters are updated so the
next append_string lands at exactly row rows.
§Errors
Returns Err(InvalidInput) if rows > self.len(). All other errors
come from underlying I/O or set_len syscalls.
Sourcepub fn reader(&self) -> Result<VariableColumnReader, Error>
pub fn reader(&self) -> Result<VariableColumnReader, Error>
Open a read-only, positionally-reading view over this column’s data +
offsets files (#56 Direction B). The returned VariableColumnReader
shares both files via independent (try_cloned) descriptors, so a single
&mut self writer can keep appending while many &self readers read
concurrently without a lock. See FixedColumnReader for the model.