Struct deltalake::DeltaTable

source ·
pub struct DeltaTable {
    pub state: Option<DeltaTableState>,
    pub config: DeltaTableConfig,
    /* private fields */
}
Expand description

The deltalake crate is currently just a meta-package shim for deltalake-core In memory representation of a Delta Table

Fields§

§state: Option<DeltaTableState>

The state of the table as of the most recent loaded Delta log entry.

§config: DeltaTableConfig

the load options used during load

Implementations§

source§

impl DeltaTable

source

pub fn new(log_store: Arc<dyn LogStore>, config: DeltaTableConfig) -> DeltaTable

Create a new Delta Table struct without loading any data from backing storage.

NOTE: This is for advanced users. If you don’t know why you need to use this method, please call one of the open_table helper methods instead.

source

pub fn object_store(&self) -> Arc<dyn ObjectStore>

get a shared reference to the delta object store

source

pub fn table_uri(&self) -> String

The URI of the underlying data

source

pub fn log_store(&self) -> Arc<dyn LogStore>

get a shared reference to the log store

source

pub async fn get_latest_version(&self) -> Result<i64, DeltaTableError>

returns the latest available version of the table

source

pub fn version(&self) -> i64

Currently loaded version of the table

source

pub async fn load(&mut self) -> Result<(), DeltaTableError>

Load DeltaTable with data from latest checkpoint

source

pub async fn update(&mut self) -> Result<(), DeltaTableError>

Updates the DeltaTable to the most recent state committed to the transaction log by loading the last checkpoint and incrementally applying each version since.

source

pub async fn peek_next_commit( &self, current_version: i64 ) -> Result<PeekCommit, DeltaTableError>

Get the list of actions for the next commit

source

pub async fn update_incremental( &mut self, max_version: Option<i64> ) -> Result<(), DeltaTableError>

Updates the DeltaTable to the latest version by incrementally applying newer versions. It assumes that the table is already updated to the current version self.version.

source

pub async fn load_version( &mut self, version: i64 ) -> Result<(), DeltaTableError>

Loads the DeltaTable state for the given version.

source

pub async fn history( &self, limit: Option<usize> ) -> Result<Vec<CommitInfo>, DeltaTableError>

Returns provenance information, including the operation, user, and so on, for each write to a table. The table history retention is based on the logRetentionDuration property of the Delta Table, 30 days by default. If limit is given, this returns the information of the latest limit commits made to this table. Otherwise, it returns all commits from the earliest commit.

source

pub fn get_active_add_actions_by_partitions<'a>( &'a self, filters: &'a [PartitionFilter] ) -> Result<impl Iterator<Item = Result<LogicalFile<'a>, DeltaTableError>> + 'a, DeltaTableError>

Obtain Add actions for files that match the filter

source

pub fn get_files_by_partitions( &self, filters: &[PartitionFilter] ) -> Result<Vec<Path>, DeltaTableError>

Returns the file list tracked in current table state filtered by provided PartitionFilters.

source

pub fn get_file_uris_by_partitions( &self, filters: &[PartitionFilter] ) -> Result<Vec<String>, DeltaTableError>

Return the file uris as strings for the partition(s)

source

pub fn get_files_iter( &self ) -> Result<impl Iterator<Item = Path>, DeltaTableError>

Returns an iterator of file names present in the loaded state

source

pub fn get_file_uris( &self ) -> Result<impl Iterator<Item = String>, DeltaTableError>

Returns a URIs for all active files present in the current table version.

source

pub fn get_files_count(&self) -> usize

Get the number of files in the table - retrn 0 if no metadata is loaded

source

pub fn snapshot(&self) -> Result<&DeltaTableState, DeltaTableError>

Returns the currently loaded state snapshot.

source

pub fn protocol(&self) -> Result<&Protocol, DeltaTableError>

Returns current table protocol

source

pub fn metadata(&self) -> Result<&Metadata, DeltaTableError>

Returns the metadata associated with the loaded state.

source

pub fn get_app_transaction_version(&self) -> HashMap<String, i64>

Returns the current version of the DeltaTable based on the loaded metadata.

source

pub fn schema(&self) -> Option<&StructType>

Return table schema parsed from transaction log. Return None if table hasn’t been loaded or no metadata was found in the log.

source

pub fn get_schema(&self) -> Result<&StructType, DeltaTableError>

Return table schema parsed from transaction log. Return DeltaTableError if table hasn’t been loaded or no metadata was found in the log.

source

pub async fn load_with_datetime( &mut self, datetime: DateTime<Utc> ) -> Result<(), DeltaTableError>

Time travel Delta table to the latest version that’s created at or before provided datetime argument.

Internally, this methods performs a binary search on all Delta transaction logs.

Trait Implementations§

source§

impl AsRef<DeltaTable> for DeltaOps

source§

fn as_ref(&self) -> &DeltaTable

Converts this type into a shared reference of the (usually inferred) input type.
source§

impl Clone for DeltaTable

source§

fn clone(&self) -> DeltaTable

Returns a copy of the value. Read more
1.0.0 · source§

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source. Read more
source§

impl Debug for DeltaTable

source§

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter. Read more
source§

impl<'de> Deserialize<'de> for DeltaTable

source§

fn deserialize<D>( deserializer: D ) -> Result<DeltaTable, <D as Deserializer<'de>>::Error>
where D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer. Read more
source§

impl Display for DeltaTable

source§

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter. Read more
source§

impl From<DeltaOps> for DeltaTable

source§

fn from(ops: DeltaOps) -> DeltaTable

Converts to this type from the input type.
source§

impl From<DeltaTable> for DeltaOps

source§

fn from(table: DeltaTable) -> DeltaOps

Converts to this type from the input type.
source§

impl Serialize for DeltaTable

source§

fn serialize<S>( &self, serializer: S ) -> Result<<S as Serializer>::Ok, <S as Serializer>::Error>
where S: Serializer,

Serialize this value into the given Serde serializer. Read more
source§

impl TableProvider for DeltaTable

source§

fn as_any(&self) -> &(dyn Any + 'static)

Returns the table provider as Any so that it can be downcast to a specific implementation.
source§

fn schema(&self) -> Arc<Schema>

Get a reference to the schema for this table
source§

fn table_type(&self) -> TableType

Get the type of this table for metadata/catalog purposes.
source§

fn get_table_definition(&self) -> Option<&str>

Get the create statement used to create this table, if available.
source§

fn get_logical_plan(&self) -> Option<&LogicalPlan>

Get the LogicalPlan of this table, if available
source§

fn scan<'life0, 'life1, 'life2, 'life3, 'async_trait>( &'life0 self, session: &'life1 SessionState, projection: Option<&'life2 Vec<usize>>, filters: &'life3 [Expr], limit: Option<usize> ) -> Pin<Box<dyn Future<Output = Result<Arc<dyn ExecutionPlan>, DataFusionError>> + Send + 'async_trait>>
where 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait, 'life3: 'async_trait, DeltaTable: 'async_trait,

Create an ExecutionPlan for scanning the table with optionally specified projection, filter and limit, described below. Read more
source§

fn supports_filter_pushdown( &self, _filter: &Expr ) -> Result<TableProviderFilterPushDown, DataFusionError>

👎Deprecated since 20.0.0: use supports_filters_pushdown instead
Tests whether the table provider can make use of a filter expression to optimise data retrieval.
source§

fn statistics(&self) -> Option<Statistics>

Get statistics for this table, if available
source§

fn constraints(&self) -> Option<&Constraints>

Get a reference to the constraints of the table. Returns: Read more
source§

fn get_column_default(&self, _column: &str) -> Option<&Expr>

Get the default value for a column, if available.
source§

fn supports_filters_pushdown( &self, filters: &[&Expr] ) -> Result<Vec<TableProviderFilterPushDown>, DataFusionError>

Tests whether the table provider can make use of any or all filter expressions to optimise data retrieval.
source§

fn insert_into<'life0, 'life1, 'async_trait>( &'life0 self, _state: &'life1 SessionState, _input: Arc<dyn ExecutionPlan>, _overwrite: bool ) -> Pin<Box<dyn Future<Output = Result<Arc<dyn ExecutionPlan>, DataFusionError>> + Send + 'async_trait>>
where 'life0: 'async_trait, 'life1: 'async_trait, Self: 'async_trait,

Return an ExecutionPlan to insert data into this table, if supported. Read more

Auto Trait Implementations§

Blanket Implementations§

source§

impl<T> Any for T
where T: 'static + ?Sized,

source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
source§

impl<T> Borrow<T> for T
where T: ?Sized,

source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
source§

impl<T> From<T> for T

source§

fn from(t: T) -> T

Returns the argument unchanged.

source§

impl<T> Instrument for T

source§

fn instrument(self, span: Span) -> Instrumented<Self>

Instruments this type with the provided Span, returning an Instrumented wrapper. Read more
source§

fn in_current_span(self) -> Instrumented<Self>

Instruments this type with the current Span, returning an Instrumented wrapper. Read more
source§

impl<T, U> Into<U> for T
where U: From<T>,

source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

source§

impl<T> Same for T

§

type Output = T

Should always be Self
source§

impl<T> ToOwned for T
where T: Clone,

§

type Owned = T

The resulting type after obtaining ownership.
source§

fn to_owned(&self) -> T

Creates owned data from borrowed data, usually by cloning. Read more
source§

fn clone_into(&self, target: &mut T)

Uses borrowed data to replace owned data, usually by cloning. Read more
source§

impl<T> ToString for T
where T: Display + ?Sized,

source§

default fn to_string(&self) -> String

Converts the given value to a String. Read more
source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

§

type Error = Infallible

The type returned in the event of a conversion error.
source§

fn try_from(value: U) -> Result<T, <T as TryFrom<U>>::Error>

Performs the conversion.
source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.
source§

impl<V, T> VZip<V> for T
where V: MultiLane<T>,

source§

fn vzip(self) -> V

source§

impl<T> WithSubscriber for T

source§

fn with_subscriber<S>(self, subscriber: S) -> WithDispatch<Self>
where S: Into<Dispatch>,

Attaches the provided Subscriber to this type, returning a WithDispatch wrapper. Read more
source§

fn with_current_subscriber(self) -> WithDispatch<Self>

Attaches the current default Subscriber to this type, returning a WithDispatch wrapper. Read more
source§

impl<T> DeserializeOwned for T
where T: for<'de> Deserialize<'de>,

source§

impl<T> Ungil for T
where T: Send,