Struct MigrationJob 

pub struct MigrationJob {
    pub table: TableIdent,
    pub old_column: String,
    pub new_column: String,
    pub text_column: String,
    pub embed_fn: EmbedFn,
    pub strategy: MigrationStrategy,
    pub batch_size: usize,
    pub new_model: Option<EmbeddingModelInfo>,
    pub on_progress: Option<ProgressFn>,
}

Migrates embedding columns in an AI-Lake table to a new model.

Usage:

let job = MigrationJob {
    table: TableIdent::new("default", "docs"),
    old_column: "embedding".to_string(),
    new_column: "embedding_v2".to_string(),
    text_column: "chunk_text".to_string(),
    embed_fn: Arc::new(|texts| Ok(my_model.encode(texts))),
    strategy: MigrationStrategy::DualWriteThenCutover,
    batch_size: 10_000,
    new_model: Some(EmbeddingModelInfo::new("my-model-v2")),
    on_progress: None,
};
job.run(catalog, store).await?;

Fields

table: TableIdent

old_column: String

Name of the embedding column to replace (e.g., “embedding”).

new_column: String

Name to give the new embedding column (e.g., “embedding_v2”). Can be the same as old_column to do an in-place model upgrade.

text_column: String

Column in the Parquet files that holds the text to re-embed. Defaults to chunk_text (the LlmContextSchema canonical name).

embed_fn: EmbedFn

Callable that converts a slice of texts to embeddings. Must return exactly texts.len() vectors, all of the same dimension.

strategy: MigrationStrategy

batch_size: usize

How many rows to embed per embed_fn call. Tune based on model batch size.

new_model: Option<EmbeddingModelInfo>

Metadata for the new embedding model — stored in Iceberg properties after migration.

on_progress: Option<ProgressFn>

Optional callback called after each file completes.

Implementations

impl MigrationJob

pub async fn run( self, catalog: Arc<dyn CatalogProvider>, store: Arc<dyn Store>, ) -> AilakeResult<()>