Expand description
Embedding model migration for AI-Lake tables.
Reads all chunks from a table, re-embeds them with a new model, and writes new files with the updated embedding column. Two strategies are supported:
AtomicReplace: replaces each file one at a time. Lower peak storage, but during the migration window different shards may have different columns.DualWriteThenCutover: writes new files containing both old and new columns, then atomically replaces all old files. Higher peak storage, zero downtime.
Structs§
- Migration
Job - Migrates embedding columns in an AI-Lake table to a new model.
- Migration
Progress - Progress reported via
on_progresscallback.
Enums§
- Migration
Strategy - How files are replaced during migration.