Struct StorageOptions

pub struct StorageOptions {
    pub backend: StorageBackendConfig,
    pub min_storage_bytes: Option<usize>,
    pub min_step_storage_bytes: Option<usize>,
    pub compression: StorageCompression,
    pub cache_mib: Option<usize>,
}

Storage configuration for a pipeline.

Fields

backend: StorageBackendConfig

How to connect to the underlying storage.

min_storage_bytes: Option<usize>

For a batch of data maintained as part of a persistent index during a pipeline run, the minimum estimated number of bytes to write it to storage.

This is provided for debugging and fine-tuning and should ordinarily be left unset.

A value of 0 will write even empty batches to storage, and nonzero values provide a threshold. usize::MAX would effectively disable storage for such batches. The default is 1,048,576 (1 MiB).

min_step_storage_bytes: Option<usize>

For a batch of data passed through the pipeline during a single step, the minimum estimated number of bytes to write it to storage.

This is provided for debugging and fine-tuning and should ordinarily be left unset. A value of 0 will write even empty batches to storage, and nonzero values provide a threshold. usize::MAX, the default, effectively disables storage for such batches. If it is set to another value, it should ordinarily be greater than or equal to min_storage_bytes.

compression: StorageCompression

The form of compression to use in data batches.

Compression has a CPU cost but it can take better advantage of limited NVMe and network bandwidth, which means that it can increase overall performance.

cache_mib: Option<usize>

The maximum size of the in-memory storage cache, in MiB.

If set, the specified cache size is spread across all the foreground and background threads. If unset, each foreground or background thread cache is limited to 256 MiB.

Trait Implementations

impl Clone for StorageOptions

fn clone(&self) -> StorageOptions

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for StorageOptions

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for StorageOptions

fn default() -> StorageOptions

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for StorageOptions

where StorageOptions: Default,

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl PartialEq for StorageOptions

fn eq(&self, other: &StorageOptions) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Serialize for StorageOptions

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl<'__s> ToSchema<'__s> for StorageOptions

fn schema() -> (&'__s str, RefOr<Schema>)

Return a tuple of name and schema or reference to a schema that can be referenced by the name or inlined directly to responses, request bodies or parameters.

fn aliases() -> Vec<(&'__s str, Schema)>

Optional set of alias schemas for the ToSchema::schema.

impl Eq for StorageOptions

impl StructuralPartialEq for StorageOptions