Struct deltalake::DeltaTableLoadOptions

pub struct DeltaTableLoadOptions {
    pub table_uri: String,
    pub storage_backend: Option<(Arc<dyn ObjectStore>, Url)>,
    pub version: DeltaVersion,
    pub require_tombstones: bool,
    pub require_files: bool,
    pub log_buffer_size: usize,
    pub log_batch_size: usize,
}

The deltalake crate is currently just a meta-package shim for deltalake-core Load-time delta table configuration options

Fields

table_uri: String

table root uri

storage_backend: Option<(Arc<dyn ObjectStore>, Url)>

backend to access storage system

version: DeltaVersion

specify the version we are going to load: a time stamp, a version, or just the newest available version

require_tombstones: bool

Indicates whether our use case requires tracking tombstones. This defaults to true

Read-only applications never require tombstones. Tombstones are only required when writing checkpoints, so even many writers may want to skip them.

require_files: bool

Indicates whether DeltaTable should track files. This defaults to true

Some append-only applications might have no need of tracking any files. Hence, DeltaTable will be loaded with significant memory reduction.

log_buffer_size: usize

Controls how many files to buffer from the commit log when updating the table. This defaults to 4 * number of cpus

Setting a value greater than 1 results in concurrent calls to the storage api. This can be helpful to decrease latency if there are many files in the log since the last checkpoint, but will also increase memory usage. Possible rate limits of the storage backend should also be considered for optimal performance.

log_batch_size: usize

Control the number of records to read / process from the commit / checkpoint files when processing record batches.

Implementations

impl DeltaTableLoadOptions

pub fn new(table_uri: impl Into<String>) -> DeltaTableLoadOptions

create default table load options for a table uri

Trait Implementations

impl Debug for DeltaTableLoadOptions

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.