Enum DeltaTableTransactionMode 

pub enum DeltaTableTransactionMode {
    None,
    Snapshot,
    Always,
    Catchup,
}

Delta table transaction mode.

Determines how the connector breaks up its input into transactions.

• none - the connector does not break up its input into transactions.
• snapshot - ingest the initial snapshot of the table in one or several transactions. If the connector is configured in the snapshot_and_follow mode, it will ingest the transaction log of the table without transactions.
• catchup - batch the initial snapshot and transaction-log catch-up into Feldera transactions (see below).
• always - all updates generated by the connector are broken up into transactions, both during the initial snapshot and when following the transaction log.

How table snapshot is ingested using transactions

If the connector is configured in the snapshot or snapshot_and_follow mode, and its transaction_mode is set to snapshot, catchup, or always, it will ingest the snapshot of the table in one or several transactions. The exact behavior depends on the value of the timestamp_column option. If timestamp_column is not set, catchup and always ingest the snapshot in one Feldera transaction; snapshot mode uses the same one-transaction behavior when timestamp_column is not set.

If timestamp_column is set, the connector will ingest the snapshot of the table in a series of batches, one for each timestamp range of width equal to the LATENESS attribute of the timestamp_column. Each range will be ingested in a separate transaction.

See timestamp_column documentation for more details.

How transaction log is ingested using transactions

If the connector is configured in the follow, snapshot_and_follow, or cdc mode, and its transaction_mode is set to catchup, it ingests the transaction log in batches. When it starts a Feldera transaction, it reads the latest available version of the Delta table (capped by end_version if set) and ingests all log entries up to and including that version in a single Feldera transaction before committing. It then repeats for subsequent versions as they appear in the log.

The catchup mode offers the most efficient way for the connector to process the entire contents of the table during backfill and keep up with table changes in steady state.

If transaction_mode is set to always, the connector ingests the transaction log in a series of transactions, generating exactly one Feldera transaction for each entry in the table’s transaction log. Feldera transaction boundaries then match the transaction boundaries of the Delta Lake table: as long as the table keeps changing, each log entry initiates a new Feldera transaction. Use always when that one-to-one alignment is required.

JSON schema

{
 "description": "Delta table transaction mode.\n\nDetermines how the connector breaks up its input into transactions.\n\n* `none` - the connector does not break up its input into transactions.\n* `snapshot` - ingest the initial snapshot of the table in one or several transactions. If the connector is\nconfigured in the `snapshot_and_follow` mode, it will ingest the transaction log of the table without transactions.\n* `catchup` - batch the initial snapshot and transaction-log catch-up into Feldera transactions (see below).\n* `always` - all updates generated by the connector are broken up into transactions, both during the initial snapshot\nand when following the transaction log.\n\n# How table snapshot is ingested using transactions\n\nIf the connector is configured in the `snapshot` or `snapshot_and_follow` mode, and its\n`transaction_mode` is set to `snapshot`, `catchup`, or `always`, it will ingest the snapshot of the\ntable in one or several transactions. The exact behavior depends on the value of the `timestamp_column` option.\nIf `timestamp_column` is not set, `catchup` and `always` ingest the snapshot in one Feldera transaction; `snapshot`\nmode uses the same one-transaction behavior when `timestamp_column` is not set.\n\nIf `timestamp_column` is set, the connector will ingest the snapshot of the table in a series of batches, one for\neach timestamp range of width equal to the `LATENESS` attribute of the `timestamp_column`. Each range will be\ningested in a separate transaction.\n\nSee `timestamp_column` documentation for more details.\n\n# How transaction log is ingested using transactions\n\nIf the connector is configured in the `follow`, `snapshot_and_follow`, or `cdc` mode, and its\n`transaction_mode` is set to `catchup`, it ingests the transaction log in batches. When it starts a\nFeldera transaction, it reads the latest available version of the Delta table (capped by `end_version` if set) and\ningests all log entries up to and including that version in a single Feldera transaction before committing. It then\nrepeats for subsequent versions as they appear in the log.\n\nThe `catchup` mode offers the most efficient way for the connector to process the entire contents\nof the table during backfill and keep up with table changes in steady state.\n\nIf `transaction_mode` is set to `always`, the connector ingests the transaction log in a series of transactions,\ngenerating exactly one Feldera transaction for each entry in the table's transaction log. Feldera transaction\nboundaries then match the transaction boundaries of the Delta Lake table: as long as the table keeps changing, each log\nentry initiates a new Feldera transaction. Use `always` when that one-to-one alignment is required.",
 "type": "string",
 "enum": [
   "none",
   "snapshot",
   "always",
   "catchup"
 ]
}

Variants

None

Snapshot

Always

Catchup

Trait Implementations

impl Clone for DeltaTableTransactionMode

fn clone(&self) -> DeltaTableTransactionMode

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for DeltaTableTransactionMode

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for DeltaTableTransactionMode

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl From<&DeltaTableTransactionMode> for DeltaTableTransactionMode

fn from(value: &DeltaTableTransactionMode) -> Self

Converts to this type from the input type.

impl FromStr for DeltaTableTransactionMode

type Err = ConversionError

The associated error which can be returned from parsing.

fn from_str(value: &str) -> Result<Self, ConversionError>

Parses a string s to return a value of this type.

impl Hash for DeltaTableTransactionMode

fn hash<__H: Hasher>(&self, state: &mut __H)

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H)

where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher.

impl Ord for DeltaTableTransactionMode

fn cmp(&self, other: &DeltaTableTransactionMode) -> Ordering

This method returns an Ordering between self and other.

fn max(self, other: Self) -> Self

where Self: Sized,

Compares and returns the maximum of two values.

fn min(self, other: Self) -> Self

where Self: Sized,

Compares and returns the minimum of two values.

fn clamp(self, min: Self, max: Self) -> Self

where Self: Sized,

Restrict a value to a certain interval.

impl PartialEq for DeltaTableTransactionMode

fn eq(&self, other: &DeltaTableTransactionMode) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl PartialOrd for DeltaTableTransactionMode

fn partial_cmp(&self, other: &DeltaTableTransactionMode) -> Option<Ordering>

This method returns an ordering between self and other values if one exists.

fn lt(&self, other: &Rhs) -> bool

Tests less than (for self and other) and is used by the < operator.

fn le(&self, other: &Rhs) -> bool

Tests less than or equal to (for self and other) and is used by the <= operator.

fn gt(&self, other: &Rhs) -> bool

Tests greater than (for self and other) and is used by the > operator.

fn ge(&self, other: &Rhs) -> bool

Tests greater than or equal to (for self and other) and is used by the >= operator.

impl Serialize for DeltaTableTransactionMode

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl ToString for DeltaTableTransactionMode

fn to_string(&self) -> String

Converts the given value to a String.

impl TryFrom<&String> for DeltaTableTransactionMode

type Error = ConversionError

The type returned in the event of a conversion error.

fn try_from(value: &String) -> Result<Self, ConversionError>

Performs the conversion.

impl TryFrom<&str> for DeltaTableTransactionMode

type Error = ConversionError

The type returned in the event of a conversion error.

fn try_from(value: &str) -> Result<Self, ConversionError>

Performs the conversion.

impl TryFrom<String> for DeltaTableTransactionMode

type Error = ConversionError

The type returned in the event of a conversion error.

fn try_from(value: String) -> Result<Self, ConversionError>

Performs the conversion.

impl Copy for DeltaTableTransactionMode

impl Eq for DeltaTableTransactionMode

impl StructuralPartialEq for DeltaTableTransactionMode