Struct IngestExternalFileOptions 

pub struct IngestExternalFileOptions { /* private fields */ }

For configuring external files ingestion.

Examples

Move files instead of copying them:

use rust_rocksdb::{DB, IngestExternalFileOptions, SstFileWriter, Options};

let writer_opts = Options::default();
let mut writer = SstFileWriter::create(&writer_opts);
let tempdir = tempfile::Builder::new()
    .tempdir()
    .expect("Failed to create temporary folder for the _path_for_sst_file");
let path1 = tempdir.path().join("_path_for_sst_file");
writer.open(path1.clone()).unwrap();
writer.put(b"k1", b"v1").unwrap();
writer.finish().unwrap();

let tempdir2 = tempfile::Builder::new()
    .prefix("_path_for_rocksdb_storageY3")
    .tempdir()
    .expect("Failed to create temporary path for the _path_for_rocksdb_storageY3");
let path2 = tempdir2.path();
{
  let db = DB::open_default(&path2).unwrap();
  let mut ingest_opts = IngestExternalFileOptions::default();
  ingest_opts.set_move_files(true);
  db.ingest_external_file_opts(&ingest_opts, vec![path1]).unwrap();
}
let _ = DB::destroy(&Options::default(), path2);

Implementations

impl IngestExternalFileOptions

pub fn set_move_files(&mut self, v: bool)

Can be set to true to move the files instead of copying them.

pub fn set_snapshot_consistency(&mut self, v: bool)

If set to false, an ingested file keys could appear in existing snapshots that where created before the file was ingested.

pub fn set_allow_global_seqno(&mut self, v: bool)

If set to false, IngestExternalFile() will fail if the file key range overlaps with existing keys or tombstones in the DB.

pub fn set_allow_blocking_flush(&mut self, v: bool)

If set to false and the file key range overlaps with the memtable key range (memtable flush required), IngestExternalFile will fail.

pub fn set_ingest_behind(&mut self, v: bool)

Set to true if you would like duplicate keys in the file being ingested to be skipped rather than overwriting existing data under that key. Usecase: back-fill of some historical data in the database without over-writing existing newer version of data. This option could only be used if the DB has been running with allow_ingest_behind=true since the dawn of time. All files will be ingested at the bottommost level with seqno=0.

pub fn get_allow_blocking_flush(&self) -> bool

Normally (true), IngestExternalFile() will trigger and block for flushing memtable(s) if there is overlap between ingested files and memtable(s). If allow_blocking_flush is set to false, IngestExternalFile() will fail if the file key range overlaps with the memtable key range (memtable flush required).

pub fn set_allow_db_generated_files(&mut self, val: bool)

EXPERIMENTAL, SUBJECT TO CHANGE

Enables special mode of ingestion that allows files generated by a live DB, instead of SstFileWriter. When true:

• Allows files to be ingested when their cf_id doesn’t match the CF they are being ingested into.
• Allows files with any sequence numbers to be ingested.
• Original sequence numbers are preserved (no reassignment).

REQUIREMENTS:

• Ingested files must NOT overlap with any existing data in the DB. Since no sequence number reassignment is performed on db generated files. Ingestion will fail if any overlap is detected. However, input files are allowed to overlap with each other when this option is enabled. This is useful when ingesting multiple levels of files from a CF, where levels naturally overlap with each other.
• CAUTION: If input files overlap with each other, then for any given user key appearing in multiple files, earlier files MUST have smaller sequence numbers than later files. Later files will be placed at a higher level (smaller level number). This is to ensure the LSM invariant where for the same key, recent updates are in higher levels. This means that if you are ingesting files from multiple levels of a CF, you should put files from lower levels first, and files from higher levels later. Example for getting files from a CF for ingestion:

ColumnFamilyMetaData cf_meta; from_db->GetColumnFamilyMetaData(from_cf, &cf_meta); // iterate in reverse to start from lowest level for (auto level_meta = cf_meta.levels.rbegin(); level_meta != cf_meta.levels.rend(); ++level_meta) { // L0 files need to be added in reverse order so we iterate in reverse // within a level too for (auto file_meta = level_meta->files.rbegin(); file_meta != level_meta->files.rend(); ++file_meta) { // Add file for ingestion } }

WARNING: Violating the sequence number ordering requirement will cause LSM invariant violations and may lead to incorrect reads or data corruption.

• If you would like to enforce that the ingested files do not overlap with each other, you can set fail_if_not_bottommost_level to true. If ingested files overlap with each other, some file will be placed above Lmax, failing the ingestion if the option is set.
• write_global_seqno must be false (sequence numbers cannot be reassigned).

pub fn get_allow_db_generated_files(&self) -> bool

Returns the value of the allow_db_generated_files option.

pub fn get_allow_global_seqno(&self) -> bool

Enables assiging a global sequence number to each ingested file, i.e., all keys in the ingested file will be treated as having this seqno. If set to false, we will use the sequence numbers in the ingested file as is, and IngestExternalFile() will fail if the ingested key range overlaps with existing keys or tombstones or output of ongoing compaction in the CF (the conditions under which a global seqno must be assigned to the ingested file). If the ingested files overlap with each other, we need to assign global sequence to the ingested files and this option needs to be enabled. One exception to this is when ingesting DB generated SST files (see option allow_db_generated_files below). DB generated files do not support global seqno assignment and can be ingested even if they overlap with each other. This option has no effect when allow_db_generated_files is enabled.

pub fn get_fail_if_not_bottommost_level(&self) -> bool

Set to TRUE if user wants file to be ingested to the last level. An error of Status::TryAgain() will be returned if a file cannot fit in the last level when calling DB::IngestExternalFile()/DB::IngestExternalFiles(). The user should clear the last level in the overlapping range before re-attempt.

ingest_behind takes precedence over fail_if_not_bottommost_level.

XXX: “bottommost” is obsolete/confusing terminology to refer to last level

pub fn set_failed_move_fall_back_to_copy(&mut self, val: bool)

If set to true, ingestion falls back to copy when hard linking fails. This applies to both move_files and link_files.

pub fn get_failed_move_fall_back_to_copy(&self) -> bool

Returns the value of the failed_move_fall_back_to_copy option.

pub fn set_file_opening_threads(&mut self, val: c_int)

Maximum number of threads used to open table readers for the files being ingested during commit, can speed up ingestion performance, when ingesting multiple files at once.

pub fn get_file_opening_threads(&self) -> c_int

Returns the value of the file_opening_threads option.

pub fn set_fill_cache(&mut self, val: bool)

Should the “data block”/“index block” read for this iteration be placed in block cache? Callers may wish to set this field to false for bulk scans. This would help not to the change eviction order of existing items in the block cache.

pub fn get_fill_cache(&self) -> bool

Returns the value of the fill_cache option.

pub fn get_ingest_behind(&self) -> bool

Set to true if you would like duplicate keys in the file being ingested to be skipped rather than overwriting existing data under that key. Use case: back-fill of some historical data in the database without over-writing existing newer version of data. This option could only be used if the CF has been running with cf_allow_ingest_behind=true since CF creation (or before any write). All files will be ingested at the bottommost level with seqno=0.

pub fn set_link_files(&mut self, val: bool)

Same as move_files except that input files will NOT be unlinked. Only one of move_files and link_files can be set at the same time.

pub fn get_link_files(&self) -> bool

Returns the value of the link_files option.

pub fn get_move_files(&self) -> bool

Can be set to true to move the files instead of copying them. The input files will be unlinked after successful ingestion. The implementation depends on hard links (LinkFile) instead of traditional move (RenameFile) to maximize the chances to restore to the original state upon failure.

pub fn set_prefetch_lmax_index_and_filter_blocks(&mut self, val: bool)

Controls whether external file ingestion should prefetch index and filter blocks while opening table readers during commit. Setting this to false can reduce commit latency for bulk loads into Lmax when (BlockBasedTableOptions::cache_index_and_filter_blocks=true or partitioned filters/indexes are enabled).

pub fn get_prefetch_lmax_index_and_filter_blocks(&self) -> bool

Returns the value of the prefetch_lmax_index_and_filter_blocks option.

pub fn get_snapshot_consistency(&self) -> bool

If set to false, an ingested file keys could appear in existing snapshots that where created before the file was ingested.

pub fn set_verify_checksums_before_ingest(&mut self, val: bool)

Set to true if you would like to verify the checksums of each block of the external SST file before ingestion. Warning: setting this to true causes slowdown in file ingestion because the external SST file has to be read.

pub fn get_verify_checksums_before_ingest(&self) -> bool

Returns the value of the verify_checksums_before_ingest option.

pub fn set_verify_checksums_readahead_size(&mut self, val: usize)

When verify_checksums_before_ingest = true, RocksDB uses default readahead setting to scan the file while verifying checksums before ingestion. Users can override the default value using this option. Using a large readahead size (> 2MB) can typically improve the performance of forward iteration on spinning disks.

pub fn get_verify_checksums_readahead_size(&self) -> usize

Returns the value of the verify_checksums_readahead_size option.

pub fn set_verify_file_checksum(&mut self, val: bool)

Set to TRUE if user wants to verify the sst file checksum of ingested files. The DB checksum function will generate the checksum of each ingested file (if file_checksum_gen_factory is set) and compare the checksum function name and checksum with the ingested checksum information.

If this option is set to True: 1) if DB does not enable checksum (file_checksum_gen_factory == nullptr), the ingested checksum information will be ignored; 2) If DB enable the checksum function, we calculate the sst file checksum after the file is moved or copied and compare the checksum and checksum name. If checksum or checksum function name does not match, ingestion will be failed. If the verification is successful, checksum and checksum function name will be stored in Manifest. If this option is set to FALSE, 1) if DB does not enable checksum, the ingested checksum information will be ignored; 2) if DB enable the checksum, we only verify the ingested checksum function name and we trust the ingested checksum. If the checksum function name matches, we store the checksum in Manifest. DB does not calculate the checksum during ingestion. However, if no checksum information is provided with the ingested files, DB will generate the checksum and store in the Manifest.

pub fn get_verify_file_checksum(&self) -> bool

Returns the value of the verify_file_checksum option.

pub fn set_write_global_seqno(&mut self, val: bool)

DEPRECATED - Set to true if you would like to write global_seqno to the external SST file on ingestion for backward compatibility before RocksDB 5.16.0. Such old versions of RocksDB expect any global_seqno to be written to the SST file rather than recorded in the DB manifest. This functionality was deprecated because (a) random writes might be costly or unsupported on some FileSystems, and (b) the file checksum changes with such a write.

pub fn get_write_global_seqno(&self) -> bool

Returns the value of the write_global_seqno option.

pub fn set_fail_if_not_bottommost_level(&mut self, val: bool)

Set to TRUE if user wants file to be ingested to the last level. An error of Status::TryAgain() will be returned if a file cannot fit in the last level when calling DB::IngestExternalFile()/DB::IngestExternalFiles(). The user should clear the last level in the overlapping range before re-attempt.

ingest_behind takes precedence over fail_if_not_bottommost_level.

XXX: “bottommost” is obsolete/confusing terminology to refer to last level.

Trait Implementations

impl AsRawPtr<rocksdb_ingestexternalfileoptions_t> for IngestExternalFileOptions

Available on crate feature raw-ptr only.

unsafe fn as_raw_ptr(&self) -> *mut rocksdb_ingestexternalfileoptions_t

Returns a raw pointer to the underlying rocksdb_ingestexternalfileoptions_t object.

This allows direct access to the RocksDB Ingest External File Options C API for advanced use cases.

impl Default for IngestExternalFileOptions

fn default() -> Self

Returns the “default value” for a type.

impl Drop for IngestExternalFileOptions

fn drop(&mut self)

Executes the destructor for this type.

fn pin_drop(self: Pin<&mut Self>)

🔬This is a nightly-only experimental API. (pin_ergonomics)

Execute the destructor for this type, but different to Drop::drop, it requires self to be pinned.

impl Send for IngestExternalFileOptions

impl Sync for IngestExternalFileOptions