Struct CompactionOptions 

pub struct CompactionOptions {

    pub target_rows_per_fragment: usize,
    pub max_rows_per_group: usize,
    pub max_bytes_per_file: Option<usize>,
    pub materialize_deletions: bool,
    pub materialize_deletions_threshold: f32,
    pub num_threads: Option<usize>,
    pub batch_size: Option<usize>,
    pub io_buffer_size: Option<u64>,
    pub defer_index_remap: bool,
    pub index_remap_mode: IndexRemapMode,
    pub compaction_mode: Option<CompactionMode>,
    pub enable_binary_copy: bool,
    pub enable_binary_copy_force: bool,
    pub binary_copy_read_batch_bytes: Option<usize>,
    pub max_source_fragments: Option<usize>,
    pub max_overlays_per_fragment: Option<usize>,
    pub transaction_properties: Option<Arc<HashMap<String, String>>>,
}

Options to be passed to compact_files.

Fields

target_rows_per_fragment: usize

Target number of rows per file. Defaults to 1 million.

This is used to determine which fragments need compaction, as any fragments that have fewer rows than this value will be candidates for compaction.

max_rows_per_group: usize

Max number of rows per group

This does not affect which fragments need compaction, but does affect how they are re-written if selected.

max_bytes_per_file: Option<usize>

Max number of bytes per file

This does not affect which frgamnets need compaction, but does affect how they are re-written if selected.

If not specified then the default (see WriteParams) will be used.

materialize_deletions: bool

Whether to compact fragments with deletions so there are no deletions. Defaults to true.

materialize_deletions_threshold: f32

The fraction of rows that need to be deleted in a fragment before materializing the deletions. Defaults to 10% (0.1). Setting to zero (or lower) will materialize deletions for all fragments with deletions. Setting above 1.0 will never materialize deletions.

num_threads: Option<usize>

The number of threads to use (how many compaction tasks to run in parallel). Defaults to the number of compute-intensive CPUs. Not used when running tasks manually using plan_compaction

batch_size: Option<usize>

The batch size to use when scanning the input fragments. If not specified then the default (see crate::dataset::Scanner::batch_size) will be used.

io_buffer_size: Option<u64>

The number of bytes to allow to queue up in the I/O buffer when scanning the input fragments. If not specified then the default (see crate::dataset::Scanner::io_buffer_size) will be used.

Increasing this can avoid a deadlock that occurs when a single batch of data is larger than the I/O buffer size.

defer_index_remap: bool

Whether to defer remapping indices during compaction. If true, indices will not be remapped during this compaction operation. Instead, the fragment reuse index is updated and will be used to perform remapping later.

index_remap_mode: IndexRemapMode

How the old-to-new row-address mapping used to remap indices is built. Defaults to IndexRemapMode::Direct.

compaction_mode: Option<CompactionMode>

The compaction mode to use. When set, this takes priority over the deprecated enable_binary_copy and enable_binary_copy_force fields.

Defaults to None (falls back to legacy boolean fields).

enable_binary_copy: bool

👎Deprecated:

Use compaction_mode instead

Deprecated: use compaction_mode instead.

enable_binary_copy_force: bool

👎Deprecated:

Use compaction_mode instead

Deprecated: use compaction_mode instead.

binary_copy_read_batch_bytes: Option<usize>

The batch size in bytes for reading during binary copy operations. Controls how much data is read at once when performing binary copy. Defaults to 16MB (16 * 1024 * 1024).

max_source_fragments: Option<usize>

Maximum number of source fragments to compact in a single run. When set, tasks are included in the plan until adding the next task would exceed this limit. This allows for incremental compaction (e.g., compact 20 fragments at a time). Defaults to None (no limit, all eligible fragments are compacted).

max_overlays_per_fragment: Option<usize>

Maximum number of data overlay files a fragment may carry before it is fully compacted. When set, any fragment with more than this many overlays is rewritten into a fresh fragment with its overlays (and deletions) materialized into the base data, dropping the fragment from any index left stale by those overlays. Defaults to Some(10). Set to Some(0) to compact every fragment that carries any overlay, or None to disable the overlay-count trigger entirely.

transaction_properties: Option<Arc<HashMap<String, String>>>

Transaction properties to store with this commit.

These key-value pairs are stored in the transaction file and can be read later to identify the source of the commit (e.g., job_id for tracking completed compaction jobs).

Implementations

impl CompactionOptions

pub fn from_dataset_config( config: &HashMap<String, String>, ) -> Result<CompactionOptions, Error>

Create CompactionOptions by starting with defaults and applying any overrides found in the dataset manifest config.

Config keys are prefixed with lance.compaction. and map to fields:

• lance.compaction.target_rows_per_fragment
• lance.compaction.max_rows_per_group
• lance.compaction.max_bytes_per_file
• lance.compaction.materialize_deletions
• lance.compaction.materialize_deletions_threshold
• lance.compaction.defer_index_remap
• lance.compaction.index_remap_mode
• lance.compaction.batch_size
• lance.compaction.io_buffer_size
• lance.compaction.compaction_mode
• lance.compaction.binary_copy_read_batch_bytes
• lance.compaction.max_source_fragments
• lance.compaction.max_overlays_per_fragment

pub fn apply_dataset_config( &mut self, config: &HashMap<String, String>, ) -> Result<(), Error>

Apply overrides from the dataset manifest config to this options struct.

Only fields with corresponding config keys are modified; other fields retain their current values.

pub fn validate(&mut self)

pub fn compaction_mode(&self) -> CompactionMode

Returns the effective CompactionMode, preferring the new compaction_mode field and falling back to the deprecated boolean fields for backwards compatibility.

pub fn transaction_properties( self, properties: HashMap<String, String>, ) -> CompactionOptions

Set transaction properties to store in the commit manifest.

Trait Implementations

impl Clone for CompactionOptions

fn clone(&self) -> CompactionOptions

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for CompactionOptions

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl Default for CompactionOptions

fn default() -> CompactionOptions

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for CompactionOptions

fn deserialize<__D>( __deserializer: __D, ) -> Result<CompactionOptions, <__D as Deserializer<'de>>::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl PartialEq for CompactionOptions

fn eq(&self, other: &CompactionOptions) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl Serialize for CompactionOptions

fn serialize<__S>( &self, __serializer: __S, ) -> Result<<__S as Serializer>::Ok, <__S as Serializer>::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl StructuralPartialEq for CompactionOptions