Struct OutputBufferConfig

pub struct OutputBufferConfig {
    pub enable_output_buffer: bool,
    pub max_output_buffer_time_millis: usize,
    pub max_output_buffer_size_records: usize,
}

Fields

enable_output_buffer: bool

Enable output buffering.

The output buffering mechanism allows decoupling the rate at which the pipeline pushes changes to the output transport from the rate of input changes.

By default, output updates produced by the pipeline are pushed directly to the output transport. Some destinations may prefer to receive updates in fewer bigger batches. For instance, when writing Parquet files, producing one bigger file every few minutes is usually better than creating small files every few milliseconds.

To achieve such input/output decoupling, users can enable output buffering by setting the enable_output_buffer flag to true. When buffering is enabled, output updates produced by the pipeline are consolidated in an internal buffer and are pushed to the output transport when one of several conditions is satisfied:

• data has been accumulated in the buffer for more than max_output_buffer_time_millis milliseconds.
• buffer size exceeds max_output_buffer_size_records records.

This flag is false by default.

max_output_buffer_time_millis: usize

Maximum time in milliseconds data is kept in the output buffer.

By default, data is kept in the buffer indefinitely until one of the other output conditions is satisfied. When this option is set the buffer will be flushed at most every max_output_buffer_time_millis milliseconds.

NOTE: this configuration option requires the enable_output_buffer flag to be set.

max_output_buffer_size_records: usize

Maximum number of updates to be kept in the output buffer.

This parameter bounds the maximal size of the buffer. Note that the size of the buffer is not always equal to the total number of updates output by the pipeline. Updates to the same record can overwrite or cancel previous updates.

By default, the buffer can grow indefinitely until one of the other output conditions is satisfied.

NOTE: this configuration option requires the enable_output_buffer flag to be set.

Implementations

impl OutputBufferConfig

pub fn validate(&self) -> Result<(), String>

Trait Implementations

impl Clone for OutputBufferConfig

fn clone(&self) -> OutputBufferConfig

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for OutputBufferConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for OutputBufferConfig

fn default() -> Self

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for OutputBufferConfig

where OutputBufferConfig: Default,

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl PartialEq for OutputBufferConfig

fn eq(&self, other: &OutputBufferConfig) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Serialize for OutputBufferConfig

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl<'__s> ToSchema<'__s> for OutputBufferConfig

fn schema() -> (&'__s str, RefOr<Schema>)

Return a tuple of name and schema or reference to a schema that can be referenced by the name or inlined directly to responses, request bodies or parameters.

fn aliases() -> Vec<(&'__s str, Schema)>

Optional set of alias schemas for the ToSchema::schema.

impl Eq for OutputBufferConfig

impl StructuralPartialEq for OutputBufferConfig