Struct Transaction 

pub struct Transaction {
    pub read_version: u64,
    pub uuid: String,
    pub operation: Operation,
    pub tag: Option<String>,
    pub transaction_properties: Option<Arc<HashMap<String, String>>>,
}

A change to a dataset that can be retried

This contains enough information to be able to build the next manifest, given the current manifest.

Fields

read_version: u64

The version of the table this transaction is based off of. If this is the first transaction, this should be 0.

uuid: String

operation: Operation

tag: Option<String>

transaction_properties: Option<Arc<HashMap<String, String>>>

Implementations

impl Transaction

pub fn new_from_version(read_version: u64, operation: Operation) -> Self

pub fn new(read_version: u64, operation: Operation, tag: Option<String>) -> Self

impl Transaction

pub fn rewritten_physical_columns( operation: &Operation, schema: &Schema, previous_fragments: &[Fragment], ) -> Vec<(u64, Vec<u32>)>

Withdraw what an in-place column rewrite invalidates from every index on a table with a tagged fragment reuse history.

The physical columns operation rewrites in place, per live fragment id, unexpanded: an update with fields_modified rewrites those fields in every updated fragment; a merge rewrites, in each fragment present in previous_fragments, the fields whose backing data file changed (merge_rewritten_fields); a data replacement rewrites the fields its new files carry, read through schema. Any other operation rewrites nothing. previous_fragments is the caller’s “before” list: the current manifest’s for a commit, the read version’s for a rebase.

pub fn withdraw_in_place_rewrites( indices: &mut Vec<IndexMetadata>, rewrites: Vec<(u64, Vec<u32>)>, schema: &Schema, live: &RoaringBitmap, lineage: Option<&TaggedLineage>, )

The one implementation of an in-place column rewrite’s effect on index coverage, shared by the commit preparation (on the list read from the current manifest) and by the conflict resolver rebasing a new index over a committed rewrite (on the index being created).

rewrites comes from Self::rewritten_physical_columns. With a walkable tagged history (lineage), every rewritten field id is expanded to its descendants through schema (a packed struct is one physical column while an index on s.x records the child’s id) and the coverage is withdrawn along the lineage (Self::withdraw_rewritten_coverage). Without a tagged history the rewritten fragments are pruned from the covering indices by id, the ordinary rule, exactly as before.

pub fn withdraw_rewritten_coverage( indices: &mut Vec<IndexMetadata>, lineage: Option<&TaggedLineage>, live: &RoaringBitmap, rewrites: &[(u64, Vec<u32>)], )

rewrites lists each rewritten live fragment with the fields rewritten in it (expanded to their descendants, see Self::withdraw_in_place_rewrites). For every non-system segment that indexes one of those fields (keyed or carried), with bitmap B:

• a rewritten fragment named in B is removed, as on an untagged table;
• every retired fragment r of B (not in live) whose rows the history moves into the rewritten fragment (destinations_of, transitively, so mixed provenance holding intermediates is covered) is removed. The sources of one transition all reach the same destinations, so the transition’s sources go whole: the segment derives no coverage of its destinations and those rows are scanned until optimize_indices rebuilds them;
• without a walkable history every retired fragment of B goes;
• a segment without a bitmap is untouched: its coverage is unknown, not empty, and the reader already excludes it.

What a withdrawal leaves is then settled per logical index name:

• coverage partly left: the segment stays and serves what is left, provided the reader will still translate it. The file keeps the withdrawn rows; the translating load drops them (they resolve outside the derived coverage), but a segment whose remaining bitmap is live-only and disjoint from the history is loaded as it is, and would serve the withdrawn retired rows raw. So when a RETIRED fragment was withdrawn and the remainder would be loaded as it is, the remainder is withdrawn too (the segment empties and is rebuilt). Withdrawing only live fragments leaves the remainder alone: rows of fragments that still exist are masked by the ordinary per-segment ownership filter;
• coverage emptied while a same-name sibling still has coverage (or an unknown one): the segment’s metadata leaves the manifest in this commit; its files are reclaimed by the ordinary index cleanup once no retained version references them;
• coverage emptied and it is the last segment of its name: it stays as the record of what index to build (queries never use it, since the reader lists nothing for it; maintenance rebuilds it from the live fragments).

Every withdrawal is logged at warn level.

pub fn prune_updated_fields_from_indices( indices: &mut [IndexMetadata], updated_fragments: &[Fragment], fields_modified: &[u32], )

If an operation modifies one or more fields in a fragment then we need to remove that fragment from any indices that cover one of the modified fields.

pub fn merge_rewritten_fields( prev_fragments: &[Fragment], new_fragments: &[Fragment], ) -> Vec<(u64, Vec<u32>)>

The columns a Merge rewrote in place: for each fragment present in both lists, the fields still present whose backing data file path changed. Brand-new fragments carry nothing stale.

impl Transaction

pub async fn restore_old_manifest( object_store: &ObjectStore, commit_handler: &dyn CommitHandler, base_path: &Path, version: u64, config: &ManifestBuildConfig, tx_path: &str, current_manifest: &Manifest, ) -> Result<(Manifest, Vec<IndexMetadata>)>

pub fn logical_index_segments(indices: &[IndexMetadata]) -> LogicalIndexSegments

Every non-system logical index, mapped to what determines its coverage.

A logical index may be backed by several physical segments, so “did this index change” is a question about the whole set. Sorted by UUID so the two sides compare positionally.

pub fn apply_mem_wal_index_coverage( final_indices: &mut [IndexMetadata], segments_before: &LogicalIndexSegments, read_version_state: Option<ReadVersionState<'_>>, new_version: u64, ) -> Result<()>

Apply MemWAL index-coverage rules once the final index list is known.

Coverage records that a base-table index contains the rows a compaction copied in, and the WAL pod retires SSTables against it.

It is derived, not reported. An index covering every fragment live at the transaction’s read version holds every row compaction had copied in by then, so it is caught up to that version’s compacted_sstables. That is the only proof available: nothing maps a generation to the fragments its rows landed in, so covering the table as the transaction read it is how an index shows it covered those rows. Fragments appended since are a later gap.

Deriving rather than transmitting means no claim can go stale between inspection and commit, the answer survives rebase (read_version is fixed for a transaction’s life), and any operation can earn coverage – an ordinary reindex that fully covers no longer has to throw its work away and wait for a repair.

Only meaningful once catch-up is required, where a missing entry means “not caught up” and the SSTables stay. A legacy table reads a missing entry as “fully caught up”, so this leaves it untouched rather than making the table look more covered than it is.

pub fn withdraw_coverage_invalidated_after_build( indices: &mut [IndexMetadata], changed: &[String], new_version: u64, ) -> Result<()>

Drop coverage for indices a post-build_manifest step narrowed.

The derivation runs while the manifest is being built, but the index list is not final there: migrate_indices can recalculate a segment’s fragment bitmap and keep its UUID, so an index can narrow after its position was decided. It reports which ones it touched rather than the caller re-snapshotting every bitmap to find out. Only ever removes.

pub fn prepare_indices( &self, current_manifest: Option<&Manifest>, current_indices: Vec<IndexMetadata>, config: &ManifestBuildConfig, frag_reuse_ledger: Option<Arc<FragReuseLedger>>, frag_reuse: FragReuseUpdate, ) -> Result<PreparedIndices>

Step one of a commit attempt: what the operation settles about the index list, computed against the manifest current at the attempt.

Runs the tagged-table write gate (system_index::frag_reuse::gate) before anything is derived, keeps current_indices as the original list, and derives the prepared list: on an in-place column rewrite (an update with fields_modified, a merge swapping a column’s data file, a data replacement) the coverage it invalidates is withdrawn along a tagged fragment reuse history or pruned by fragment id on any other table (withdraw_in_place_rewrites). Every other operation prepares the list unchanged; Restore never reaches the builder and restores its snapshot’s list whole.

frag_reuse_ledger is the decoded history of the current entry when the caller read it (the commit path decodes it once per attempt for the operations above); without it an inline history is decoded from the entry and an external one is refused, so a commit never guesses at the lineage. A history that does not decode is an error, never treated as “no lineage”. frag_reuse is what the commit path settled about the entry itself.

pub fn build_manifest( &self, current_manifest: Option<&Manifest>, current_indices: Vec<IndexMetadata>, transaction_file_path: &str, config: &ManifestBuildConfig, ) -> Result<(Manifest, Vec<IndexMetadata>)>

Create a new manifest from the current manifest and the transaction.

current_manifest should only be None if the dataset does not yet exist.

pub fn build_manifest_with_read_version( &self, current_manifest: Option<&Manifest>, current_indices: Vec<IndexMetadata>, transaction_file_path: &str, config: &ManifestBuildConfig, read_version_state: Option<ReadVersionState<'_>>, ) -> Result<(Manifest, Vec<IndexMetadata>)>

Self::build_manifest with the version this transaction read.

Supplied by the commit path, which already materializes that version. None where there is none to read – dataset creation and detached commits – in which case no index can be shown to cover it and coverage is left as the invalidation rules put it.

Prepares the index list itself with nothing settled about the fragment reuse entry and no history read, so it stays safe on its own: a tagged entry on a Rewrite or a CreateIndex touching a tagged entry is refused, an inline history is decoded from the entry and an external one is refused. The commit path prepares first (prepare_indices) and builds with build_manifest_prepared.

pub fn build_manifest_prepared( &self, current_manifest: Option<&Manifest>, prepared: PreparedIndices, transaction_file_path: &str, config: &ManifestBuildConfig, read_version_state: Option<ReadVersionState<'_>>, ) -> Result<(Manifest, Vec<IndexMetadata>)>

Build the next manifest from a prepared index list (prepare_indices): the builder applies the operation to the prepared list and publishes only the final list; the original list is what MemWAL coverage is compared against. The prepared result must have been derived against current_manifest.

Trait Implementations

impl Clone for Transaction

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Transaction

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl DeepSizeOf for Transaction

fn deep_size_of_children(&self, __context: &mut Context) -> usize

fn deep_size_of(&self) -> usize

impl From<&Transaction> for Transaction

fn from(value: &Transaction) -> Self

Converts to this type from the input type.

impl From<&Transaction> for Transaction

fn from(value: &Transaction) -> Self

Converts to this type from the input type.

impl PartialEq for Transaction

fn eq(&self, other: &Self) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for Transaction

impl TryFrom<Transaction> for Transaction

type Error = Error

The type returned in the event of a conversion error.

fn try_from(message: Transaction) -> Result<Self>

Performs the conversion.