Skip to main content

Module migration

Module migration 

Source
Expand description

Embedding model migration for AI-Lake tables.

Reads all chunks from a table, re-embeds them with a new model, and writes new files with the updated embedding column. Two strategies are supported:

  • AtomicReplace: replaces each file one at a time. Lower peak storage, but during the migration window different shards may have different columns.
  • DualWriteThenCutover: writes new files containing both old and new columns, then atomically replaces all old files. Higher peak storage, zero downtime.

Structs§

MigrationJob
Migrates embedding columns in an AI-Lake table to a new model.
MigrationProgress
Progress reported via on_progress callback.

Enums§

MigrationStrategy
How files are replaced during migration.

Type Aliases§

EmbedFn
ProgressFn