Struct DataFileEntry 

pub struct DataFileEntry {

    pub path: String,
    pub record_count: u64,
    pub file_size_bytes: u64,
    pub centroid_b64: Option<String>,
    pub radius: Option<f32>,
    pub hnsw_offset: Option<u64>,
    pub hnsw_len: Option<u64>,
    pub vector_column: Option<String>,
    pub vector_dim: Option<u32>,
    pub extra_vector_indexes: Vec<ExtraVectorIndex>,
    pub index_status: IndexStatus,
    pub index_error: Option<String>,
    pub batch_id: Option<String>,
    pub embedding_model: Option<String>,
    pub partition_value: Option<String>,
    pub deletion_vector: Option<DeletionVector>,
    pub first_row_id: Option<i64>,
    pub column_stats: Option<String>,
}

Metadata about a single data file in a table snapshot.

Fields

path: String

Relative path within the warehouse (e.g., “data/part-00001.parquet”)

record_count: u64

file_size_bytes: u64

centroid_b64: Option<String>

base64-encoded centroid F32 values (primary vector column)

radius: Option<f32>

hnsw_offset: Option<u64>

hnsw_len: Option<u64>

vector_column: Option<String>

vector_dim: Option<u32>

extra_vector_indexes: Vec<ExtraVectorIndex>

Additional vector columns beyond the primary (empty for single-column tables).

index_status: IndexStatus

Index build status. Defaults to Ready for backward compatibility with old manifests.

index_error: Option<String>

Human-readable error from a failed background index build. Set only when index_status == IndexStatus::Failed; None otherwise.

batch_id: Option<String>

Caller-supplied idempotency key. When set, write_batch_idempotent skips the write if a file with the same batch_id is already committed in the snapshot.

embedding_model: Option<String>

Embedding model identifier stored per-file so mixed-model tables (during migration) can be identified without reading the main metadata.json.

partition_value: Option<String>

Partition value for this file (e.g. the agent_id UUID). Written per-file when VectorStoragePolicy::partition_by is set. Enables manifest-level pruning: search skips files whose partition_value doesn’t match the requested partition filter, avoiding all HNSW I/O.

deletion_vector: Option<DeletionVector>

Iceberg V3 Deletion Vector: Roaring Bitmap of deleted row positions. None for V2 tables or V3 tables with no deletes for this file. When present, scanner masks these row IDs from HNSW results.

first_row_id: Option<i64>

Iceberg V3 Row Lineage: globally unique first row ID assigned to this file. Computed at commit time from the table’s cumulative next-row-id counter. None for V2 tables (row lineage requires format-version=3).

column_stats: Option<String>

JSON-encoded BTreeMap<i32, FieldStats> (column_stats::extract_column_stats), computed once from this file’s own Parquet footer at write/compaction time. write_manifest_file decodes this to emit real value_counts/ null_value_counts/column_sizes/lower_bounds/upper_bounds Avro fields instead of null, enabling row-group pruning for Spark/Trino/DuckDB reading the table as plain Iceberg. None for files predating this feature, or where extraction failed — always spec-safe since these fields are optional at every format version.

Implementations

impl DataFileEntry

pub fn is_foreign(&self) -> bool

True iff this file was never written by the AI-Lake SDK — most likely rewritten by a generic Iceberg engine (Spark/Trino OPTIMIZE/rewrite_data_files, DuckDB) with no knowledge of AI-Lake.

Every AI-Lake write path (make_data_file_entry, make_data_file_entry_indexing) computes and stores a centroid unconditionally, even for IndexStatus::Indexing files — before the HNSW build itself completes. A missing centroid is therefore a reliable “not ours” signal, distinct from IndexStatus::Failed (an internal indexing failure on a file the SDK did write — see writer.rs::patch_index_failed, which never touches centroid_b64). Shared by CompactionPlanner::plan(), scanner.rs::search, and ailake info so the three can’t drift on the definition.

pub fn batch_ids(&self) -> Vec<String>

Decode batch_id into the set of original idempotency keys it covers.

A freshly-written file’s batch_id is a plain string (the caller-supplied key). A file produced by compaction (merge_batch_ids) instead carries a JSON-encoded array of every source file’s key(s), since one merged file can supersede several idempotent writes at once. This tries the JSON-array form first and falls back to treating the raw string as a single key — the two encodings share the same Option<String> field so this is the only place that needs to know the difference. Returns [] when batch_id is None.

pub fn merge_batch_ids(sources: &[DataFileEntry]) -> Option<String>

Build the batch_id value for a file merging sources (compaction).

Flattens every source’s batch_ids() (not just source.batch_id verbatim — a source may itself already be a previous merge, so this must decode before re-encoding to avoid nesting JSON arrays deeper on every subsequent compaction pass), dedups while preserving first-seen order, and re-encodes as a JSON array. Returns None when no source carried a batch_id — compaction of files that were never written idempotently produces a plain, key-less file, same as before this method existed.

Trait Implementations

impl Clone for DataFileEntry

fn clone(&self) -> DataFileEntry

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for DataFileEntry

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for DataFileEntry

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Serialize for DataFileEntry

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.