Struct Fragment 

pub struct Fragment {
    pub id: u64,
    pub files: Vec<DataFile>,
    pub overlays: Vec<DataOverlayFile>,
    pub deletion_file: Option<DeletionFile>,
    pub row_id_meta: Option<RowIdMeta>,
    pub physical_rows: Option<usize>,
    pub last_updated_at_version_meta: Option<RowDatasetVersionMeta>,
    pub created_at_version_meta: Option<RowDatasetVersionMeta>,
}

Data fragment.

A fragment is a set of files which represent the different columns of the same rows. If column exists in the schema, but the related file does not exist, treat this column as nulls.

Fields

id: u64

Fragment ID

files: Vec<DataFile>

Files within the fragment.

overlays: Vec<DataOverlayFile>

Overlay files supplying new values for a subset of cells without rewriting the base data files. Order is significant: a later entry is newer than an earlier one. See DataOverlayFile for resolution rules.

deletion_file: Option<DeletionFile>

Optional file with deleted local row offsets.

row_id_meta: Option<RowIdMeta>

RowIndex

physical_rows: Option<usize>

Original number of rows in the fragment. If this is None, then it is unknown. This is only optional for legacy reasons. All new tables should have this set.

last_updated_at_version_meta: Option<RowDatasetVersionMeta>

Last updated at version metadata

created_at_version_meta: Option<RowDatasetVersionMeta>

Created at version metadata

Implementations

impl Fragment

pub fn new(id: u64) -> Fragment

pub fn num_rows(&self) -> Option<usize>

pub fn referenced_lance_files(&self) -> impl Iterator<Item = &DataFile>

Every Lance-format file this fragment references: the base data files plus the data file of each overlay. The fragment’s other referenced files are not in this format: a deletion file is .arrow or .bin, and external row-id or row-version metadata is a raw byte range.

Prefer this over files, which is the base data files only and so omits overlays.

pub fn referenced_lance_files_mut( &mut self, ) -> impl Iterator<Item = &mut DataFile>

Mutable counterpart of Self::referenced_lance_files, for rewriting the fields a clone has to normalize (base_id) across base and overlay files alike.

pub fn has_spilled_row_lineage(&self) -> bool

Whether any of this fragment’s row lineage sequences lives in a data file column rather than inline.

pub fn row_lineage_file( &self, field_id: i32, ) -> Result<Option<&DataFile>, Error>

The data file holding the row lineage column with the reserved field_id, which is the one entry of Self::files whose fields carry it. None when no file does, which for a sequence whose metadata says it is spilled is corruption; so is more than one file carrying the id, which this reports as an error.

pub fn spilled_row_lineage_field_ids(&self) -> Vec<i32>

The reserved field ids of the row lineage sequences this fragment marks as spilled: ROW_ID_FIELD_ID when its row ids are, and likewise ROW_CREATED_AT_VERSION_FIELD_ID and ROW_LAST_UPDATED_AT_VERSION_FIELD_ID for its versions.

These ids are never in the dataset schema, so code that decides whether a data file is still needed by its schema fields has to keep a file carrying one of them as well: that file is the sequence’s only copy.

pub fn from_json(json: &str) -> Result<Fragment, Error>

pub fn with_file_legacy( id: u64, path: &str, schema: &Schema, physical_rows: Option<usize>, ) -> Fragment

Create a Fragment with one DataFile

pub fn with_file( self, path: impl Into<String>, field_ids: Vec<i32>, column_indices: Vec<i32>, version: ConcreteFileVersion, file_size_bytes: Option<NonZero<u64>>, ) -> Fragment

pub fn with_physical_rows(self, physical_rows: usize) -> Fragment

pub fn add_file( &mut self, path: impl Into<String>, field_ids: Vec<i32>, column_indices: Vec<i32>, version: ConcreteFileVersion, file_size_bytes: Option<NonZero<u64>>, )

pub fn add_file_legacy(&mut self, path: &str, schema: &Schema)

Add a new DataFile to this fragment.

pub fn try_infer_version( fragments: &[Fragment], ) -> Result<Option<ConcreteFileVersion>, Error>

Trait Implementations

impl Clone for Fragment

fn clone(&self) -> Fragment

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Fragment

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl DeepSizeOf for Fragment

fn deep_size_of_children(&self, __context: &mut Context) -> usize

fn deep_size_of(&self) -> usize

impl<'de> Deserialize<'de> for Fragment

fn deserialize<__D>( __deserializer: __D, ) -> Result<Fragment, <__D as Deserializer<'de>>::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Eq for Fragment

impl From<&Fragment> for DataFragment

fn from(f: &Fragment) -> DataFragment

Converts to this type from the input type.

impl From<FileFragment> for Fragment

fn from(fragment: FileFragment) -> Self

Converts to this type from the input type.

impl PartialEq for Fragment

fn eq(&self, other: &Fragment) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl Serialize for Fragment

fn serialize<__S>( &self, __serializer: __S, ) -> Result<<__S as Serializer>::Ok, <__S as Serializer>::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl StructuralPartialEq for Fragment

impl TryFrom<DataFragment> for Fragment

type Error = Error

The type returned in the event of a conversion error.

fn try_from(p: DataFragment) -> Result<Fragment, Error>

Performs the conversion.