Struct CompactionOptions 

pub struct CompactionOptions {

    pub target_rows_per_fragment: usize,
    pub max_rows_per_group: usize,
    pub max_bytes_per_file: Option<usize>,
    pub materialize_deletions: bool,
    pub materialize_deletions_threshold: f32,
    pub num_threads: Option<usize>,
    pub batch_size: Option<usize>,
    pub io_buffer_size: Option<u64>,
    pub defer_index_remap: bool,
    pub index_remap_mode: IndexRemapMode,
    pub compaction_mode: Option<CompactionMode>,
    pub enable_binary_copy: bool,
    pub enable_binary_copy_force: bool,
    pub binary_copy_read_batch_bytes: Option<usize>,
    pub max_source_fragments: Option<usize>,
    pub max_source_rows: Option<usize>,
    pub max_source_bytes: Option<u64>,
    pub excluded_fragment_ids: Vec<u32>,
    pub max_overlays_per_fragment: Option<usize>,
    pub data_storage_version: Option<LanceFileVersion>,
    pub transaction_properties: Option<Arc<HashMap<String, String>>>,
}

Options to be passed to compact_files.

Fields

target_rows_per_fragment: usize

Target number of rows per file. Defaults to 1 million.

This is used to determine which fragments need compaction, as any fragments that have fewer rows than this value will be candidates for compaction.

max_rows_per_group: usize

Max number of rows per group

This does not affect which fragments need compaction, but does affect how they are re-written if selected.

max_bytes_per_file: Option<usize>

Max number of bytes per file

This does not affect which frgamnets need compaction, but does affect how they are re-written if selected.

If not specified then the default (see WriteParams) will be used.

materialize_deletions: bool

Whether to compact fragments with deletions so there are no deletions. Defaults to true.

materialize_deletions_threshold: f32

The fraction of rows that need to be deleted in a fragment before materializing the deletions. Defaults to 10% (0.1). Setting to zero (or lower) will materialize deletions for all fragments with deletions. Setting above 1.0 will never materialize deletions.

num_threads: Option<usize>

The number of threads to use (how many compaction tasks to run in parallel). Defaults to the number of compute-intensive CPUs. Not used when running tasks manually using plan_compaction

batch_size: Option<usize>

The batch size to use when scanning the input fragments. If not specified then the default (see crate::dataset::Scanner::batch_size) will be used.

io_buffer_size: Option<u64>

The number of bytes to allow to queue up in the I/O buffer when scanning the input fragments. If not specified then the default (see crate::dataset::Scanner::io_buffer_size) will be used.

Increasing this can avoid a deadlock that occurs when a single batch of data is larger than the I/O buffer size.

defer_index_remap: bool

Whether to defer remapping indices during compaction. If true, indices will not be remapped during this compaction operation. Instead, the fragment reuse index is updated and will be used to perform remapping later.

index_remap_mode: IndexRemapMode

How the old-to-new row-address mapping used to remap indices is built. Defaults to IndexRemapMode::Direct.

compaction_mode: Option<CompactionMode>

The compaction mode to use. When set, this takes priority over the deprecated enable_binary_copy and enable_binary_copy_force fields.

Defaults to None (falls back to legacy boolean fields).

enable_binary_copy: bool

👎Deprecated:

Use compaction_mode instead

Deprecated: use compaction_mode instead.

enable_binary_copy_force: bool

👎Deprecated:

Use compaction_mode instead

Deprecated: use compaction_mode instead.

binary_copy_read_batch_bytes: Option<usize>

The batch size in bytes for reading during binary copy operations. Controls how much data is read at once when performing binary copy. Defaults to 16MB (16 * 1024 * 1024).

max_source_fragments: Option<usize>

Maximum number of source fragments to compact in a single run. When set, tasks are included in the plan until adding the next task would exceed this limit. This allows for incremental compaction (e.g., compact 20 fragments at a time). Defaults to None (no limit, all eligible fragments are compacted).

max_source_rows: Option<usize>

Maximum number of source rows to compact in a single run. Rows are counted as live rows (physical rows minus soft-deleted rows). When set, tasks are included in the plan until adding the next task would exceed this limit. Defaults to None (no limit).

max_source_bytes: Option<u64>

Maximum number of source bytes to compact in a single run, measured as the total size of the source fragments’ data and overlay files. When set, tasks are included in the plan until adding the next task would exceed this limit. Blob v2 payloads live in separate blob files and are not counted, so this is not a cap on total compaction I/O for datasets with blob columns. Defaults to None (no limit).

excluded_fragment_ids: Vec<u32>

Fragment IDs to exclude from compaction planning.

Excluded fragments act as boundaries between adjacent compaction candidates, so fragments on opposite sides of an exclusion are never combined into the same task. IDs that are duplicated or absent from the dataset are ignored. Defaults to an empty list.

max_overlays_per_fragment: Option<usize>

Maximum number of data overlay files a fragment may carry before it is fully compacted. When set, any fragment with more than this many overlays is rewritten into a fresh fragment with its overlays (and deletions) materialized into the base data, dropping the fragment from any index left stale by those overlays. Defaults to Some(10). Set to Some(0) to compact every fragment that carries any overlay, or None to disable the overlay-count trigger entirely.

data_storage_version: Option<LanceFileVersion>

Exact data file version for compacted output.

If omitted, use the dataset’s default write version without changing it. The planner resolves release selectors before distributing tasks. Targets cannot cross the V1/V2 boundary.

let options = CompactionOptions {
    data_storage_version: Some(LanceFileVersion::V2_2),
    ..Default::default()
};

transaction_properties: Option<Arc<HashMap<String, String>>>

Transaction properties to store with this commit.

These key-value pairs are stored in the transaction file and can be read later to identify the source of the commit (e.g., job_id for tracking completed compaction jobs).

Implementations

impl CompactionOptions

pub fn from_dataset_config(config: &HashMap<String, String>) -> Result<Self>

Create CompactionOptions by starting with defaults and applying any overrides found in the dataset manifest config.

Config keys are prefixed with lance.compaction. and map to fields:

• lance.compaction.target_rows_per_fragment
• lance.compaction.max_rows_per_group
• lance.compaction.max_bytes_per_file
• lance.compaction.materialize_deletions
• lance.compaction.materialize_deletions_threshold
• lance.compaction.defer_index_remap
• lance.compaction.index_remap_mode
• lance.compaction.batch_size
• lance.compaction.io_buffer_size
• lance.compaction.compaction_mode
• lance.compaction.binary_copy_read_batch_bytes
• lance.compaction.max_source_fragments
• lance.compaction.max_source_rows
• lance.compaction.max_source_bytes
• lance.compaction.max_overlays_per_fragment
• lance.compaction.data_storage_version

pub fn apply_dataset_config( &mut self, config: &HashMap<String, String>, ) -> Result<()>

Apply overrides from the dataset manifest config to this options struct.

Only fields with corresponding config keys are modified; other fields retain their current values.

pub fn validate(&mut self) -> Result<()>

pub fn compaction_mode(&self) -> CompactionMode

Returns the effective CompactionMode, preferring the new compaction_mode field and falling back to the deprecated boolean fields for backwards compatibility.

pub fn transaction_properties(self, properties: HashMap<String, String>) -> Self

Set transaction properties to store in the commit manifest.

Trait Implementations

impl Clone for CompactionOptions

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for CompactionOptions

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for CompactionOptions

fn default() -> Self

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for CompactionOptions

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl PartialEq for CompactionOptions

fn eq(&self, other: &Self) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl Serialize for CompactionOptions

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl StructuralPartialEq for CompactionOptions