Struct WriteParams 

pub struct WriteParams {

    pub max_rows_per_file: usize,
    pub max_rows_per_group: usize,
    pub max_bytes_per_file: usize,
    pub mode: WriteMode,
    pub store_params: Option<ObjectStoreParams>,
    pub base_store_params: Option<HashMap<String, ObjectStoreParams>>,
    pub progress: Arc<dyn WriteFragmentProgress>,
    pub write_progress: Option<WriteProgressFn>,
    pub commit_handler: Option<Arc<dyn CommitHandler>>,
    pub data_storage_version: Option<LanceFileVersion>,
    pub enable_stable_row_ids: bool,
    pub enable_v2_manifest_paths: bool,
    pub session: Option<Arc<Session>>,
    pub auto_cleanup: Option<AutoCleanupParams>,
    pub skip_auto_cleanup: bool,
    pub transaction_properties: Option<Arc<HashMap<String, String>>>,
    pub initial_bases: Option<Vec<BasePath>>,
    pub target_bases: Option<Vec<u32>>,
    pub target_base_names_or_paths: Option<Vec<String>>,
    pub allow_external_blob_outside_bases: bool,
    pub external_blob_mode: ExternalBlobMode,
    pub blob_pack_file_size_threshold: Option<usize>,
}

Dataset Write Parameters

Fields

max_rows_per_file: usize

Max number of records per file.

max_rows_per_group: usize

Max number of rows per row group.

max_bytes_per_file: usize

Max file size in bytes.

This is a soft limit. The actual file size may be larger than this value by a few megabytes, since once we detect we hit this limit, we still need to flush the footer.

This limit is checked after writing each group, so if max_rows_per_group is set to a large value, this limit may be exceeded by a large amount.

The default is 90 GB. If you are using an object store such as S3, we currently have a hard 100 GB limit.

mode: WriteMode

Write mode

store_params: Option<ObjectStoreParams>

base_store_params: Option<HashMap<String, ObjectStoreParams>>

progress: Arc<dyn WriteFragmentProgress>

write_progress: Option<WriteProgressFn>

Optional callback invoked after each batch is written.

Receives cumulative WriteStats so callers can render a progress bar or compute throughput. The callback must be cheap and non-blocking; spawn a task if you need async work.

commit_handler: Option<Arc<dyn CommitHandler>>

If present, dataset will use this to update the latest version

If not set, the default will be based on the object store. Generally this will be RenameCommitHandler unless the object store does not handle atomic renames (e.g. S3)

If a custom object store is provided (via store_params.object_store) then this must also be provided.

data_storage_version: Option<LanceFileVersion>

The format version to use when writing data.

Newer versions are more efficient but the data can only be read by more recent versions of lance.

If not specified then the latest stable version will be used.

enable_stable_row_ids: bool

Experimental: if set to true, the writer will use stable row ids. These row ids are stable after compaction operations, but not after updates. This makes compaction more efficient, since with stable row ids no secondary indices need to be updated to point to new row ids.

enable_v2_manifest_paths: bool

If set to true, and this is a new dataset, uses the new v2 manifest paths. These allow constant-time lookups for the latest manifest on object storage. This parameter has no effect on existing datasets. To migrate an existing dataset, use the super::Dataset::migrate_manifest_paths_v2 method. Default is True.

session: Option<Arc<Session>>

auto_cleanup: Option<AutoCleanupParams>

If Some and this is a new dataset, old dataset versions will be automatically cleaned up after commits according to the parameters set out in AutoCleanupParams. This parameter has no effect on existing datasets. To add auto-cleanup to an existing dataset, use Dataset::update_config to set lance.auto_cleanup.interval and lance.auto_cleanup.older_than. Both parameters must be set to invoke auto-cleanup.

Defaults to None (auto-cleanup disabled). Enabling it makes every interval-th commit run a full cleanup pass, which lists and reads every manifest in the dataset even when nothing is old enough to delete; on object stores this adds noticeable per-commit latency that grows with the version count. Prefer calling Dataset::cleanup_old_versions explicitly when you actually need to reclaim space.

skip_auto_cleanup: bool

If true, skip auto cleanup during commits. This should be set to true for high frequency writes to improve performance. This is also useful if the writer does not have delete permissions and the clean up would just try and log a failure anyway. Default is false.

transaction_properties: Option<Arc<HashMap<String, String>>>

Configuration key-value pairs for this write operation. This can include commit messages, engine information, etc. this properties map will be persisted as part of Transaction object.

initial_bases: Option<Vec<BasePath>>

New base paths to register in the manifest during dataset creation. Each BasePath must have a properly assigned ID (non-zero). Only used in CREATE/OVERWRITE modes for manifest registration. IDs should be assigned by the caller before passing to WriteParams.

target_bases: Option<Vec<u32>>

Target base IDs for writing data files. When provided, all new data files will be written to bases with these IDs. Used in all modes (CREATE, APPEND, OVERWRITE) to specify where data should be written. The IDs must correspond to either:

• IDs in initial_bases (for CREATE/OVERWRITE modes)
• IDs already registered in the existing dataset manifest (for APPEND mode)

target_base_names_or_paths: Option<Vec<String>>

Target base names or paths as strings (unresolved). These will be resolved to IDs when the write operation executes. Resolution happens at builder execution time when dataset context is available.

allow_external_blob_outside_bases: bool

Allow writing external blob URIs that cannot be mapped to any registered non-dataset-root base path. When disabled, such rows are rejected.

external_blob_mode: ExternalBlobMode

The strategy used when writing external blob URIs.

blob_pack_file_size_threshold: Option<usize>

Maximum size in bytes for blob v2 pack (.blob) sidecar files. When a pack file reaches this size, a new one is started. If not set, defaults to 1 GiB.

Implementations

impl WriteParams

pub fn with_storage_version(version: LanceFileVersion) -> WriteParams

Create a new WriteParams with the given storage version. The other fields are set to their default values.

pub fn storage_version_or_default(&self) -> LanceFileVersion

pub fn store_registry(&self) -> Arc<ObjectStoreRegistry>

pub fn with_base_store_params( self, base_path: impl AsRef<str>, store_params: ObjectStoreParams, ) -> WriteParams

Set exact runtime object store params for a registered base path.

These params are used as-is for that base. The write-level default store_params remain the fallback for bases without an explicit binding.

pub fn with_transaction_properties( self, properties: HashMap<String, String>, ) -> WriteParams

Set the properties for this WriteParams.

pub fn with_initial_bases(self, bases: Vec<BasePath>) -> WriteParams

Set the initial_bases for this WriteParams.

This specifies new base paths to register in the manifest during dataset creation. Each BasePath must have a properly assigned ID (non-zero) before calling this method. Only used in CREATE/OVERWRITE modes for manifest registration.

pub fn with_target_bases(self, base_ids: Vec<u32>) -> WriteParams

Set the target_bases for this WriteParams.

This specifies the base IDs where data files should be written. The IDs must correspond to either:

• IDs in initial_bases (for CREATE/OVERWRITE modes)
• IDs already registered in the existing dataset manifest (for APPEND mode)

pub fn with_target_base_names_or_paths( self, references: Vec<String>, ) -> WriteParams

Store target base names or paths for deferred resolution.

This method stores the references in target_base_names_or_paths field to be resolved later at execution time when the dataset manifest is available.

Resolution will happen at write execution time and will try to match:

1. initial_bases by name
2. initial_bases by path
3. existing manifest by name
4. existing manifest by path

Arguments

• references - Vector of base names or paths to be resolved later

pub fn with_allow_external_blob_outside_bases(self, allow: bool) -> WriteParams

Configure whether external blobs outside registered bases are allowed.

pub fn with_external_blob_mode(self, mode: ExternalBlobMode) -> WriteParams

Configure how external blob URIs are handled during writes.

pub fn with_blob_pack_file_size_threshold(self, max_bytes: usize) -> WriteParams

Set the maximum size in bytes for blob v2 pack (.blob) sidecar files.

Trait Implementations

impl Clone for WriteParams

fn clone(&self) -> WriteParams

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for WriteParams

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl Default for WriteParams

fn default() -> WriteParams

Returns the “default value” for a type.