Struct WriteParams 

pub struct WriteParams {

    pub max_rows_per_file: usize,
    pub max_rows_per_group: usize,
    pub max_bytes_per_file: usize,
    pub mode: WriteMode,
    pub store_params: Option<ObjectStoreParams>,
    pub progress: Arc<dyn WriteFragmentProgress>,
    pub commit_handler: Option<Arc<dyn CommitHandler>>,
    pub data_storage_version: Option<LanceFileVersion>,
    pub enable_stable_row_ids: bool,
    pub enable_v2_manifest_paths: bool,
    pub session: Option<Arc<Session>>,
    pub auto_cleanup: Option<AutoCleanupParams>,
    pub skip_auto_cleanup: bool,
    pub transaction_properties: Option<Arc<HashMap<String, String>>>,
    pub initial_bases: Option<Vec<BasePath>>,
    pub target_bases: Option<Vec<u32>>,
    pub target_base_names_or_paths: Option<Vec<String>>,
}

Dataset Write Parameters

Fields

max_rows_per_file: usize

Max number of records per file.

max_rows_per_group: usize

Max number of rows per row group.

max_bytes_per_file: usize

Max file size in bytes.

This is a soft limit. The actual file size may be larger than this value by a few megabytes, since once we detect we hit this limit, we still need to flush the footer.

This limit is checked after writing each group, so if max_rows_per_group is set to a large value, this limit may be exceeded by a large amount.

The default is 90 GB. If you are using an object store such as S3, we currently have a hard 100 GB limit.

mode: WriteMode

Write mode

store_params: Option<ObjectStoreParams>

progress: Arc<dyn WriteFragmentProgress>

commit_handler: Option<Arc<dyn CommitHandler>>

If present, dataset will use this to update the latest version

If not set, the default will be based on the object store. Generally this will be RenameCommitHandler unless the object store does not handle atomic renames (e.g. S3)

If a custom object store is provided (via store_params.object_store) then this must also be provided.

data_storage_version: Option<LanceFileVersion>

The format version to use when writing data.

Newer versions are more efficient but the data can only be read by more recent versions of lance.

If not specified then the latest stable version will be used.

enable_stable_row_ids: bool

Experimental: if set to true, the writer will use stable row ids. These row ids are stable after compaction operations, but not after updates. This makes compaction more efficient, since with stable row ids no secondary indices need to be updated to point to new row ids.

enable_v2_manifest_paths: bool

If set to true, and this is a new dataset, uses the new v2 manifest paths. These allow constant-time lookups for the latest manifest on object storage. This parameter has no effect on existing datasets. To migrate an existing dataset, use the super::Dataset::migrate_manifest_paths_v2 method. Default is True.

session: Option<Arc<Session>>

auto_cleanup: Option<AutoCleanupParams>

If Some and this is a new dataset, old dataset versions will be automatically cleaned up according to the parameters set out in AutoCleanupParams. This parameter has no effect on existing datasets. To add auto-cleanup to an existing dataset, use Dataset::update_config to set lance.auto_cleanup.interval and lance.auto_cleanup.older_than. Both parameters must be set to invoke auto-cleanup.

skip_auto_cleanup: bool

If true, skip auto cleanup during commits. This should be set to true for high frequency writes to improve performance. This is also useful if the writer does not have delete permissions and the clean up would just try and log a failure anyway. Default is false.

transaction_properties: Option<Arc<HashMap<String, String>>>

Configuration key-value pairs for this write operation. This can include commit messages, engine information, etc. this properties map will be persisted as part of Transaction object.

initial_bases: Option<Vec<BasePath>>

New base paths to register in the manifest during dataset creation. Each BasePath must have a properly assigned ID (non-zero). Only used in CREATE/OVERWRITE modes for manifest registration. IDs should be assigned by the caller before passing to WriteParams.

target_bases: Option<Vec<u32>>

Target base IDs for writing data files. When provided, all new data files will be written to bases with these IDs. Used in all modes (CREATE, APPEND, OVERWRITE) to specify where data should be written. The IDs must correspond to either:

• IDs in initial_bases (for CREATE/OVERWRITE modes)
• IDs already registered in the existing dataset manifest (for APPEND mode)

target_base_names_or_paths: Option<Vec<String>>

Target base names or paths as strings (unresolved). These will be resolved to IDs when the write operation executes. Resolution happens at builder execution time when dataset context is available.

Implementations

impl WriteParams

pub fn with_storage_version(version: LanceFileVersion) -> Self

Create a new WriteParams with the given storage version. The other fields are set to their default values.

pub fn storage_version_or_default(&self) -> LanceFileVersion

pub fn store_registry(&self) -> Arc<ObjectStoreRegistry>

pub fn with_transaction_properties( self, properties: HashMap<String, String>, ) -> Self

Set the properties for this WriteParams.

pub fn with_initial_bases(self, bases: Vec<BasePath>) -> Self

Set the initial_bases for this WriteParams.

This specifies new base paths to register in the manifest during dataset creation. Each BasePath must have a properly assigned ID (non-zero) before calling this method. Only used in CREATE/OVERWRITE modes for manifest registration.

pub fn with_target_bases(self, base_ids: Vec<u32>) -> Self

Set the target_bases for this WriteParams.

This specifies the base IDs where data files should be written. The IDs must correspond to either:

• IDs in initial_bases (for CREATE/OVERWRITE modes)
• IDs already registered in the existing dataset manifest (for APPEND mode)

pub fn with_target_base_names_or_paths(self, references: Vec<String>) -> Self

Store target base names or paths for deferred resolution.

This method stores the references in target_base_names_or_paths field to be resolved later at execution time when the dataset manifest is available.

Resolution will happen at write execution time and will try to match:

1. initial_bases by name
2. initial_bases by path
3. existing manifest by name
4. existing manifest by path

Arguments

• references - Vector of base names or paths to be resolved later

Trait Implementations

impl Clone for WriteParams

fn clone(&self) -> WriteParams

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for WriteParams

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for WriteParams

fn default() -> Self

Returns the “default value” for a type.