Enum Operation 

pub enum Operation {
    Append {
        fragments: Vec<Fragment>,
    },
    Delete {
        updated_fragments: Vec<Fragment>,
        deleted_fragment_ids: Vec<u64>,
        predicate: String,
    },
    Overwrite {
        fragments: Vec<Fragment>,
        schema: Schema,
        config_upsert_values: Option<HashMap<String, String>>,
        initial_bases: Option<Vec<BasePath>>,
    },
    CreateIndex {
        new_indices: Vec<IndexMetadata>,
        removed_indices: Vec<IndexMetadata>,
    },
    Rewrite {
        groups: Vec<RewriteGroup>,
        rewritten_indices: Vec<RewrittenIndex>,
        frag_reuse_index: Option<IndexMetadata>,
    },
    DataReplacement {
        replacements: Vec<DataReplacementGroup>,
    },
    DataOverlay {
        groups: Vec<DataOverlayGroup>,
    },
    Merge {
        fragments: Vec<Fragment>,
        schema: Schema,
        preserves_nullability: bool,
    },
    Restore {
        version: u64,
    },
    ReserveFragments {
        num_fragments: u32,
    },
    Update {
        removed_fragment_ids: Vec<u64>,
        updated_fragments: Vec<Fragment>,
        new_fragments: Vec<Fragment>,
        fields_modified: Vec<u32>,
        compacted_sstables: Vec<CompactedSsTable>,
        fields_for_preserving_frag_bitmap: Vec<u32>,
        update_mode: Option<UpdateMode>,
        inserted_rows_filter: Option<KeyExistenceFilter>,
        updated_fragment_offsets: Option<UpdatedFragmentOffsets>,
    },
    Project {
        schema: Schema,
        preserves_nullability: bool,
    },
    UpdateConfig {
        config_updates: Option<UpdateMap>,
        table_metadata_updates: Option<UpdateMap>,
        schema_metadata_updates: Option<UpdateMap>,
        field_metadata_updates: HashMap<i32, UpdateMap>,
    },
    UpdateMemWalState {
        compacted_sstables: Vec<CompactedSsTable>,
    },
    Clone {
        is_shallow: bool,
        ref_name: Option<String>,
        ref_version: u64,
        ref_path: String,
        branch_name: Option<String>,
    },
    UpdateBases {
        new_bases: Vec<BasePath>,
    },
}

An operation on a dataset.

Variants

Append

Adding new fragments to the dataset. The fragments contained within haven’t yet been assigned a final ID.

Fields

fragments: Vec<Fragment>

Delete

Updated fragments contain those that have been modified with new deletion files. The deleted fragment IDs are those that should be removed from the manifest.

Fields

updated_fragments: Vec<Fragment>

deleted_fragment_ids: Vec<u64>

predicate: String

Overwrite

Overwrite the entire dataset with the given fragments. This is also used when initially creating a table.

The fragments are newly written ones and are assigned fresh ids at commit time, continuing from the dataset’s highest id ever used; the ids they arrive with are ignored.

A fragment carrying a deletion file is rejected. A deletion file’s path embeds the fragment id, so it cannot follow its fragment to the new id: minting a fragment and giving it a deletion file are mutually exclusive in one transaction. Use Self::Delete to commit deletions against existing fragments, or Self::Merge to change their schema.

Fields

fragments: Vec<Fragment>

schema: Schema

config_upsert_values: Option<HashMap<String, String>>

initial_bases: Option<Vec<BasePath>>

CreateIndex

A new index has been created.

Fields

new_indices: Vec<IndexMetadata>

The new secondary indices, any existing indices with the same name will be replaced.

removed_indices: Vec<IndexMetadata>

The indices that have been modified.

Rewrite

Data is rewritten but not modified. This is used for things like compaction or re-ordering. Contains the old fragments and the new ones that have been replaced.

This operation will modify the row addresses of existing rows and so any existing index covering a rewritten fragment will need to be remapped.

Fields

groups: Vec<RewriteGroup>

Groups of fragments that have been modified

rewritten_indices: Vec<RewrittenIndex>

Indices that have been updated with the new row addresses

frag_reuse_index: Option<IndexMetadata>

The fragment reuse index to be created or updated to

DataReplacement

Replace data in a column in the dataset with new data. This is used for null column population where we replace an entirely null column with a new column that has data.

This operation will only allow replacing files that contain the same schema e.g. if the original files contain columns A, B, C and the new files contain only columns A, B then the operation is not allowed. As we would need to split the original files into two files, one with column A, B and the other with column C.

Corollary to the above: the operation will also not allow replacing files unless the affected columns all have the same datafile layout across the fragments being replaced.

e.g. if fragments being replaced contain files with different schema layouts on the column being replaced, the operation is not allowed. say frag_1: [A] [B, C] and frag_2: [A, B] [C] and we are trying to replace column A with a new column A, the operation is not allowed.

Fields

replacements: Vec<DataReplacementGroup>

DataOverlay

Attach overlay files to fragments, supplying new values for a subset of (physical offset, field) cells without rewriting the fragments’ base data files. See DataOverlayFile and the Data Overlay Files specification for resolution, coverage, and versioning rules.

Fields

groups: Vec<DataOverlayGroup>

Merge

Merge a new column in ‘fragments’ is the final fragments include all data files, the new fragments must align with old ones at rows. ‘schema’ is not forced to include existed columns, which means we could use Merge to drop column data

Fields

fragments: Vec<Fragment>

schema: Schema

preserves_nullability: bool

Set when this merge makes no nullability-affecting schema change: it introduces no field that data staged against an earlier schema could not safely omit. Without the assertion the merge conflicts with concurrent appends in either commit order, since a stale append omits new columns entirely and its rows read as null.

Restore

Restore an old version of the database

Fields

version: u64

ReserveFragments

Reserves fragment ids for future use This can be used when row ids need to be known before a transaction has been committed. It is used during a rewrite operation to allow indices to be remapped to the new row ids as part of the operation.

Fields

num_fragments: u32

Update

Update values in the dataset.

Updates are generally vertical or horizontal.

A vertical update adds new rows. In this case, the updated_fragments will only have existing rows deleted and will not have any new fields added. All new data will be contained in new_fragments. This is what is used by a merge_insert that matches the whole schema and what is used by the dataset updater.

A horizontal update adds new columns. In this case, the updated fragments may have fields removed or added. It is even possible for a field to be tombstoned and then added back in the same update. (which is a field modification). If any fields are modified in this way then they need to be added to the fields_modified list. This way we can correctly update the indices. This is what is used by a merge insert that does not match the whole schema.

Fields

removed_fragment_ids: Vec<u64>

Ids of fragments that have been moved

updated_fragments: Vec<Fragment>

Fragments that have been updated

new_fragments: Vec<Fragment>

Fragments that have been added

fields_modified: Vec<u32>

The fields that have been modified

compacted_sstables: Vec<CompactedSsTable>

MemWAL SSTables to mark as compacted after this transaction.

fields_for_preserving_frag_bitmap: Vec<u32>

The fields that used to judge whether to preserve the new frag’s id into the frag bitmap of the specified indices.

update_mode: Option<UpdateMode>

The mode of update

inserted_rows_filter: Option<KeyExistenceFilter>

Optional filter for detecting conflicts on inserted row keys. Only tracks keys from INSERT operations during merge insert, not updates.

updated_fragment_offsets: Option<UpdatedFragmentOffsets>

Physical row offsets (per fragment) that matched update_columns for RewriteColumns. None means callers did not supply offsets; build_manifest skips partial refresh then.

Project

Project to a new schema.

Fields

schema: Schema

preserves_nullability: bool

Set when this projection makes no nullability-affecting schema change, as a rename or a drop does not. A nullability tightening must not set this: its producer proved the claim by scanning at its read version, so a concurrent write can falsify it and the projection conflicts with value-writes in either commit order.

UpdateConfig

Update the dataset configuration and metadata.

Schema or field metadata updates conflict with a concurrent Self::Merge in either commit order. A merge carries complete schema state from its read version, so rebasing the operations could discard metadata installed by the other transaction.

Fields

config_updates: Option<UpdateMap>

table_metadata_updates: Option<UpdateMap>

schema_metadata_updates: Option<UpdateMap>

field_metadata_updates: HashMap<i32, UpdateMap>

UpdateMemWalState

Update SSTable compaction progress in the MemWAL index.

This is used during merge-insert to atomically record which SSTables have been compacted into the base table.

Fields

compacted_sstables: Vec<CompactedSsTable>

Clone

Clone a dataset.

Fields

is_shallow: bool

ref_name: Option<String>

ref_version: u64

ref_path: String

branch_name: Option<String>

UpdateBases

Fields

new_bases: Vec<BasePath>

The new base paths to add to the manifest.

Implementations

impl Operation

pub fn modifies_same_metadata(&self, other: &Self) -> bool

pub fn upsert_key_conflict(&self, other: &Self) -> bool

Check whether another operation upserts a key that is referenced by another operation

impl Operation

pub fn name(&self) -> &str

Trait Implementations

impl Clone for Operation

fn clone(&self) -> Operation

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Operation

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl DeepSizeOf for Operation

fn deep_size_of_children(&self, __context: &mut Context) -> usize

fn deep_size_of(&self) -> usize

impl Display for Operation

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for Operation

fn eq(&self, other: &Self) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.