Skip to main content

PackedPayload

Struct PackedPayload 

Source
pub struct PackedPayload { /* private fields */ }
Expand description

One payload column: all five facts of one object packed adjacently into a single FixedSizeBinary(33) element.

This is the arm the first two were both missing. FourTables and OneTableFourColumns differ in how many tables they use and agree in having five payload columns, so a full-row fetch costs five buffer reads at five unrelated addresses in both. Here it costs one:

per row, byte for byte (little-endian, no padding, no nulls):
  0  u64  offset
  8  u64  len
 16  u8   object type code (1 commit, 2 tree, 3 blob, 4 tag, 6 ofs-delta, 7 ref-delta)
 17  u64  uncompressed_size
 25  u64  delta_base — archive offset of the base entry, 0 for none
 33  ── next row

It was 24 bytes narrower before delta_base (PLAN §13). A 33-byte record spans one 64-byte cache line 49% of the time and two the rest, so a full-row fetch is ~1.5 lines against the five the columnar arms touch — where at 25 bytes it was ~1.4 against four. The trade moved slightly, in both directions at once, which is why the numbers below were re-taken rather than scaled.

The oid stays in its own column. It is the key the stree resolves against, not one of the facts a resolved lookup fetches, so packing it in would grow every payload read by 20 bytes nobody reads at lookup time.

§The trade, stated before it is measured

  • Full-row access should win — one stride instead of four.
  • Column scans should lose, and lose badly. count_type reads one byte per row; the columnar arms stream n bytes to answer it, this arm drags 33n through cache to use n. sum_uncompressed reads 8 of every 33 instead of 8 of every 8.
  • It is no longer typed Arrow columns. A FixedSizeBinary(33) is opaque to DuckDB / Polars / DataFusion, which can query the other two arms directly off the IPC bytes. That is a real cost and it is not a performance one.

Implementations§

Source§

impl PackedPayload

Source

pub fn build_with_oid_layout( entries: &[IndexEntry], layout: OidLayout, ) -> Result<Self>

ObjectIndex::build with the stree section’s OidLayout chosen explicitly — the whole apparatus of the alignment experiment (examples/oid_align_bench.rs).

It lives on this arm alone rather than on the trait because ordinals_batch is the same code in all three arms (they share one OidResolver), so a second arm would add a second Arrow payload to keep resident and would measure nothing new. This is the arm the payload sweep picked.

Source

pub fn keyspace_phase(&self) -> usize

The cache-line phase of this index’s stree keyspace — 0 for OidLayout::Aligned64. The bench asserts on it before quoting a number, because two arms that landed on the same phase would be one arm measured twice.

Source

pub fn oid_column(&self) -> &FixedSizeBinaryArray

Source

pub fn column_is_inside_ipc(&self) -> bool

Trait Implementations§

Source§

impl ObjectIndex for PackedPayload

Source§

fn sum_uncompressed(&self) -> u64

Strided: 8 useful bytes out of every 25 touched.

Source§

fn count_type(&self, t: ObjType) -> usize

Strided: 1 useful byte out of every 25 touched.

Source§

fn build(entries: &[IndexEntry]) -> Result<Self>

Source§

fn lookup(&self, oid: &[u8]) -> Option<IndexRow>

Resolve one oid. None for an absent oid and for an oid of the wrong width.
Source§

fn lookup_batch(&self, oids: &[&[u8]]) -> Vec<Option<IndexRow>>

The batch path, and the one that matters: have negotiation sends up to 1000 oids at a time, the push connectivity check sends thousands. Positional — out[i] answers oids[i]. Read more
Source§

fn ordinals_batch(&self, oids: &[&[u8]]) -> Vec<Option<u32>>

oid → ordinal and stop. No payload column is touched at all. Read more
Source§

fn extents_batch(&self, oids: &[&[u8]]) -> Vec<Option<(u64, u64)>>

The partial-row access pattern: byte extent only, two of the five facts. This is extents(&[oid]), what the wire path actually asks for when it is about to copy bytes out of a pack.
Source§

fn name(&self) -> &'static str

Source§

fn len(&self) -> usize

Source§

fn ipc_bytes(&self) -> usize

Total bytes of Arrow IPC this index holds resident. Four sections or one, this counts the same payload, so it is comparable across arms.
Source§

fn resident_bytes(&self) -> usize

IPC bytes plus the stree oid section — everything the index keeps alive, excluding the handful of Arc’d schema/metadata allocations.
Source§

fn is_empty(&self) -> bool

Auto Trait Implementations§

Blanket Implementations§

Source§

impl<T> Allocation for T
where T: RefUnwindSafe + Send + Sync,

Source§

impl<T> Any for T
where T: 'static + ?Sized,

Source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
Source§

impl<T> Borrow<T> for T
where T: ?Sized,

Source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
Source§

impl<ST, DT> CastableFrom<ST, Initialized, Initialized> for DT
where ST: ?Sized, DT: ?Sized,

Source§

impl<ST, DT> CastableFrom<ST, Uninit, Uninit> for DT
where ST: ?Sized, DT: ?Sized,

Source§

impl<T> From<T> for T

Source§

fn from(t: T) -> T

Returns the argument unchanged.

Source§

impl<T, U> Into<U> for T
where U: From<T>,

Source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Source§

impl<T> Read<Exclusive, BecauseExclusive> for T
where T: ?Sized,

Source§

impl<T> Same for T

Source§

type Output = T

Should always be Self
Source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

type Error = Infallible

The type returned in the event of a conversion error.
Source§

fn try_from(value: U) -> Result<T, <T as TryFrom<U>>::Error>

Performs the conversion.
Source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.