pub fn filter_deleted_rows<T>(
batch: RecordBatch,
parallel: Vec<T>,
bitmap: &RoaringBitmap,
) -> AilakeResult<(RecordBatch, Vec<T>)>Expand description
Removes DV-masked rows from a Parquet-read (batch, parallel) pair, where
parallel[i] corresponds positionally to batch row i (embeddings, texts, …).
Used by rewrite jobs (compaction, migration, backfill) that read a file’s raw rows
and write them into a brand-new physical file: since the new file has fresh row
positions, a deleted row can’t just be re-masked with the old bitmap after the fact —
it must be dropped from the input before the merge, or it silently reappears (no DV
on the new DataFileEntry, or a DV bitmap that no longer lines up with the new
row order).