pub struct PackedPayload { /* private fields */ }Expand description
One payload column: all five facts of one object packed adjacently into
a single FixedSizeBinary(33) element.
This is the arm the first two were both missing. FourTables and
OneTableFourColumns differ in how many tables they use and agree in
having five payload columns, so a full-row fetch costs five buffer reads
at five unrelated addresses in both. Here it costs one:
per row, byte for byte (little-endian, no padding, no nulls):
0 u64 offset
8 u64 len
16 u8 object type code (1 commit, 2 tree, 3 blob, 4 tag, 6 ofs-delta, 7 ref-delta)
17 u64 uncompressed_size
25 u64 delta_base — archive offset of the base entry, 0 for none
33 ── next rowIt was 24 bytes narrower before delta_base (PLAN §13). A 33-byte record
spans one 64-byte cache line 49% of the time and two the rest, so a full-row
fetch is ~1.5 lines against the five the columnar arms touch — where at 25
bytes it was ~1.4 against four. The trade moved slightly, in both directions
at once, which is why the numbers below were re-taken rather than scaled.
The oid stays in its own column. It is the key the stree resolves
against, not one of the facts a resolved lookup fetches, so packing it in
would grow every payload read by 20 bytes nobody reads at lookup time.
§The trade, stated before it is measured
- Full-row access should win — one stride instead of four.
- Column scans should lose, and lose badly.
count_typereads one byte per row; the columnar arms streamnbytes to answer it, this arm drags33nthrough cache to usen.sum_uncompressedreads 8 of every 33 instead of 8 of every 8. - It is no longer typed Arrow columns. A
FixedSizeBinary(33)is opaque to DuckDB / Polars / DataFusion, which can query the other two arms directly off the IPC bytes. That is a real cost and it is not a performance one.
Implementations§
Source§impl PackedPayload
impl PackedPayload
Sourcepub fn build_with_oid_layout(
entries: &[IndexEntry],
layout: OidLayout,
) -> Result<Self>
pub fn build_with_oid_layout( entries: &[IndexEntry], layout: OidLayout, ) -> Result<Self>
ObjectIndex::build with the stree section’s OidLayout chosen
explicitly — the whole apparatus of the alignment experiment
(examples/oid_align_bench.rs).
It lives on this arm alone rather than on the trait because
ordinals_batch is the same code in all three arms (they share one
OidResolver), so a second arm would add a second Arrow payload to
keep resident and would measure nothing new. This is the arm the payload
sweep picked.
Sourcepub fn keyspace_phase(&self) -> usize
pub fn keyspace_phase(&self) -> usize
The cache-line phase of this index’s stree keyspace — 0 for
OidLayout::Aligned64. The bench asserts on it before quoting a
number, because two arms that landed on the same phase would be one arm
measured twice.
pub fn oid_column(&self) -> &FixedSizeBinaryArray
pub fn column_is_inside_ipc(&self) -> bool
Trait Implementations§
Source§impl ObjectIndex for PackedPayload
impl ObjectIndex for PackedPayload
Source§fn sum_uncompressed(&self) -> u64
fn sum_uncompressed(&self) -> u64
Strided: 8 useful bytes out of every 25 touched.
Source§fn count_type(&self, t: ObjType) -> usize
fn count_type(&self, t: ObjType) -> usize
Strided: 1 useful byte out of every 25 touched.
fn build(entries: &[IndexEntry]) -> Result<Self>
Source§fn lookup(&self, oid: &[u8]) -> Option<IndexRow>
fn lookup(&self, oid: &[u8]) -> Option<IndexRow>
None for an absent oid and for an oid of the
wrong width.Source§fn lookup_batch(&self, oids: &[&[u8]]) -> Vec<Option<IndexRow>>
fn lookup_batch(&self, oids: &[&[u8]]) -> Vec<Option<IndexRow>>
have negotiation sends up to
1000 oids at a time, the push connectivity check sends thousands.
Positional — out[i] answers oids[i]. Read moreSource§fn ordinals_batch(&self, oids: &[&[u8]]) -> Vec<Option<u32>>
fn ordinals_batch(&self, oids: &[&[u8]]) -> Vec<Option<u32>>
Source§fn extents_batch(&self, oids: &[&[u8]]) -> Vec<Option<(u64, u64)>>
fn extents_batch(&self, oids: &[&[u8]]) -> Vec<Option<(u64, u64)>>
extents(&[oid]), what the wire path actually asks for
when it is about to copy bytes out of a pack.fn name(&self) -> &'static str
Source§fn oid_at(&self, ordinal: u32) -> Vec<u8> ⓘ
fn oid_at(&self, ordinal: u32) -> Vec<u8> ⓘ
ordinal. Every arm keeps a FixedSizeBinary oid
column, so this is one bounds-checked slice, copied out — owned rather
than borrowed so that a stack holding its projection behind a lock can
answer it too. Positional, like Arrow’s own value(i): an ordinal past
len() (or, on a stack, a tail-tagged one) is a programming error and
panics rather than answering.Source§fn entry_at(&self, ordinal: u32) -> IndexEntry
fn entry_at(&self, ordinal: u32) -> IndexEntry
ordinal, as the entry that would rebuild it.
The fold’s merge reads the projection back through this.Source§fn extent_at(&self, ordinal: u32) -> (u64, u64)
fn extent_at(&self, ordinal: u32) -> (u64, u64)
(offset, len) of the row at ordinal — positional, no oid, no
allocation. The crash-recovery diff scans the whole projection through
this, so it must cost a column read and nothing more.Source§fn oids_raw(&self) -> Vec<u8> ⓘ
fn oids_raw(&self) -> Vec<u8> ⓘ
len() × oid_len bytes.
This IS the ordinal space the derived tables address (__gunnar_reach__
bitmaps, the commit graph’s token), and on a persisted arm it is the oid
column’s own bytes: the thing objects.ordinal used to cache in a file
of its own, with a header and an epoch to validate it, is a contiguous
buffer this index already holds. The default is the positional loop; the
columnar arms hand back their column.fn len(&self) -> usize
Source§fn ipc_bytes(&self) -> usize
fn ipc_bytes(&self) -> usize
Source§fn resident_bytes(&self) -> usize
fn resident_bytes(&self) -> usize
stree oid section — everything the index keeps
alive, excluding the handful of Arc’d schema/metadata allocations.fn is_empty(&self) -> bool
Source§fn persists() -> boolwhere
Self: Sized,
fn persists() -> boolwhere
Self: Sized,
from_persisted. An arm that
answers false is never persisted: the stack then keeps every row in
redb and rebuilds from it, which is the pre-2026-09-04 behaviour, kept
as a correct fallback rather than made an error. Read moreSource§fn persisted_bytes(&self) -> Option<&[u8]>
fn persisted_bytes(&self) -> Option<&[u8]>
None for an arm with no such form.Source§fn from_persisted(buf: Buffer) -> Result<Self>where
Self: Sized,
fn from_persisted(buf: Buffer) -> Result<Self>where
Self: Sized,
persisted_bytes
produced — typically a memory-mapped objects.index, so every array is a
view into the page cache rather than a copy of it. Refuses a buffer this
arm did not write (a different arm’s layout, a different format stamp).Source§fn entries(&self) -> Vec<IndexEntry>
fn entries(&self) -> Vec<IndexEntry>
len() allocations of oid, which
is the price of handing build a slice; the fold pays it once per fold
and nothing on the read path calls it.