Skip to main content

Module backfill

Module backfill 

Source
Expand description

Backfill job: adds a new vector column to existing files in an AI-Lake table.

Reads each file, generates embeddings for the new column via embed_fn, and rewrites the file with both the original vector column and the new one (using write_multi). Commits an Overwrite snapshot per file (AtomicReplace semantics).

Idempotent: files that already contain the new column (detected via has_column_footer) are skipped.

Re-exports§

pub use crate::migration::EmbedFn;

Structs§

BackfillJob
Adds a new vector column to all existing files in a table.
BackfillProgress
Progress reported after each file is backfilled.

Type Aliases§

BackfillProgressFn