Struct FileFragment 

pub struct FileFragment { /* private fields */ }

A Fragment of a Lance Dataset.

The interface is modeled after pyarrow.dataset.Fragment.

Implementations

impl FileFragment

pub fn new(dataset: Arc<Dataset>, metadata: Fragment) -> Self

Creates a new FileFragment.

pub async fn create( dataset_uri: &str, id: usize, source: impl StreamingWriteSource, params: Option<WriteParams>, ) -> Result<Fragment>

Create a new FileFragment from a StreamingWriteSource.

This method can be used before a Dataset is created. For example, Fragments can be created distributed first, before a central machine to commit the dataset with these fragments.

pub async fn create_fragments( dataset_uri: &str, source: impl StreamingWriteSource, params: Option<WriteParams>, ) -> Result<Vec<Fragment>>

Create a list of FileFragment from a StreamingWriteSource.

pub async fn create_from_file( filename: &str, dataset: &Dataset, fragment_id: usize, physical_rows: Option<usize>, ) -> Result<Fragment>

pub fn dataset(&self) -> &Dataset

pub fn schema(&self) -> &Schema

pub fn metadata(&self) -> &Fragment

Returns the fragment’s metadata.

pub fn id(&self) -> usize

The id of this FileFragment.

pub fn num_data_files(&self) -> usize

The number of data files in this fragment.

pub fn data_file_for_field(&self, field_id: u32) -> Option<&DataFile>

Gets the data file for a given field

pub async fn open( &self, projection: &Schema, read_config: FragReadConfig, ) -> Result<FragmentReader>

Open a FileFragment with a given default projection.

All read operations (other than read_projected) will use the supplied default projection. For read_projected, the projection must be a subset of the default projection.

Parameters

• projection: The projection schema.
• read_config: Controls what columns are included in the output.
• scan_scheduler: The scheduler to use for reading data files. If not supplied and the data is v2 data then a new scheduler will be created

projection may be an empty schema only if with_row_id is true. In that case, the reader will only be generating row ids.

pub async fn count_rows(&self, filter: Option<String>) -> Result<usize>

Count the rows in this fragment.

pub async fn count_deletions(&self) -> Result<usize>

Get the number of rows that have been deleted in this fragment.

pub fn fast_physical_rows(&self) -> Result<usize>

Get the number of physical rows in the fragment synchronously

Fails if the fragment does not have the physical row count in the metadata. This method should only be called in new workflows which are not run on old versions of Lance.

pub fn fast_num_deletions(&self) -> Result<usize>

Get the number of deleted rows in the fragment synchronously

Fails if the fragment does not have deletion count in the metadata. This method should only be called in new workflows which are not run on old versions of Lance.

pub fn fast_logical_rows(&self) -> Result<usize>

Get the number of logical rows (physical rows - deleted rows) in the fragment synchronously

Fails if the fragment does not have the physical row count or deletion count in the metadata. This method should only be called in new workflows which are not run on old versions of Lance.

pub async fn physical_rows(&self) -> Result<usize>

Get the number of physical rows in the fragment. This includes deleted rows.

If there are no deleted rows, this is equal to the number of rows in the fragment.

pub async fn validate(&self) -> Result<()>

Validate the fragment

Verifies:

• All field ids in the fragment are distinct
• Within each data file, field ids are in increasing order
• All data files exist and have the same length
• Field ids are distinct between data files.
• Deletion file exists and has rowids in the correct range
• Fragment.physical_rows matches length of file
• DeletionFile.num_deleted_rows matches length of deletion vector

pub async fn open_session( &self, projection: &Schema, with_row_address: bool, ) -> Result<FragmentSession>

Open a FragmentSession, which manages a short-lived session of FileFragment.

This API works well for users making repeated requests over the same projected schema.

pub async fn take( &self, indices: &[u32], projection: &Schema, ) -> Result<RecordBatch>

Take rows from this fragment based on the offset in the file.

This will always return the same number of rows as the input indices. If indices are out-of-bounds, this will return an error.

pub async fn get_deletion_vector(&self) -> Result<Option<Arc<DeletionVector>>>

Get the deletion vector for this fragment, using the cache if available.

pub fn scan(&self) -> Scanner

Scan this FileFragment.

See Dataset::scan.

pub async fn merge_columns( &mut self, stream: impl RecordBatchReader + Send + 'static, left_on: &str, right_on: &str, max_field_id: i32, ) -> Result<(Fragment, Schema)>

pub async fn update_columns( &mut self, right_stream: impl RecordBatchReader + Send + 'static, left_on: &str, right_on: &str, ) -> Result<(Fragment, Vec<u32>)>

pub async fn add_columns( &self, transforms: NewColumnTransform, read_columns: Option<Vec<String>>, batch_size: Option<u32>, ) -> Result<(Fragment, Schema)>

Append new columns to the fragment

This is the fragment-level version of Dataset::add_columns.

pub async fn delete(self, predicate: &str) -> Result<Option<Self>>

Delete rows from the fragment.

If all rows are deleted, returns Ok(None). Otherwise, returns a new fragment with the updated deletion vector. This must be persisted to the manifest.

pub async fn extend_deletions( self, new_deletions: impl IntoIterator<Item = u32>, ) -> Result<Option<Self>>

Trait Implementations

impl Clone for FileFragment

fn clone(&self) -> FileFragment

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for FileFragment

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl From<FileFragment> for Fragment

fn from(fragment: FileFragment) -> Self

Converts to this type from the input type.