internal-api only.Expand description
This module provides log replay utilities.
Log replay is the process of transforming an iterator of action batches (read from Delta transaction logs) into an iterator of filtered/transformed actions for specific use cases. The logs, which record all table changes as JSON entries, are processed batch by batch, typically from newest to oldest.
Log replay is currently implemented for table scans, which filter and apply transformations to produce file actions which builds the view of the table state at a specific point in time. Future extensions will support additional log replay processors beyond the current use case. (e.g. checkpointing: filter actions to include only those needed to rebuild table state)
This module provides structures for efficient batch processing, focusing on file action
deduplication with FileActionDeduplicator which tracks unique files across log batches
to minimize memory usage for tables with extensive history.
Structs§
- Actions
Batch - File
Action Key - The subset of file action fields that uniquely identifies it in the log, used for deduplication of adds and removes during log replay.
Traits§
- HasSelection
Vector - This trait is used to determine if a processor’s output contains any selected rows. This is used to filter out batches with no selected rows from the log replay results.
- LogReplay
Processor - A trait for processing batches of actions from Delta transaction logs during log replay.
- Parallel
LogReplay Processor