Struct RuntimeConfig

pub struct RuntimeConfig {

    pub workers: u16,
    pub storage: Option<StorageOptions>,
    pub fault_tolerance: FtConfig,
    pub cpu_profiler: bool,
    pub tracing: bool,
    pub tracing_endpoint_jaeger: String,
    pub min_batch_size_records: u64,
    pub max_buffering_delay_usecs: u64,
    pub resources: ResourceConfig,
    pub clock_resolution_usecs: Option<u64>,
    pub pin_cpus: Vec<usize>,
    pub provisioning_timeout_secs: Option<u64>,
    pub max_parallel_connector_init: Option<u64>,
    pub init_containers: Option<Value>,
}

Global pipeline configuration settings. This is the publicly exposed type for users to configure pipelines.

Fields

workers: u16

Number of DBSP worker threads.

Each DBSP “foreground” worker thread is paired with a “background” thread for LSM merging, making the total number of threads twice the specified number.

The typical sweet spot for the number of workers is between 4 and 16. Each worker increases overall memory consumption for data structures used during a step.

storage: Option<StorageOptions>

Storage configuration.

• If this is None, the default, the pipeline’s state is kept in in-memory data-structures. This is useful if the pipeline’s state will fit in memory and if the pipeline is ephemeral and does not need to be recovered after a restart. The pipeline will most likely run faster since it does not need to access storage.

• If set, the pipeline’s state is kept on storage. This allows the pipeline to work with state that will not fit into memory. It also allows the state to be checkpointed and recovered across restarts.

fault_tolerance: FtConfig

Fault tolerance configuration.

cpu_profiler: bool

Enable CPU profiler.

The default value is true.

tracing: bool

Enable pipeline tracing.

tracing_endpoint_jaeger: String

Jaeger tracing endpoint to send tracing information to.

min_batch_size_records: u64

Minimal input batch size.

The controller delays pushing input records to the circuit until at least min_batch_size_records records have been received (total across all endpoints) or max_buffering_delay_usecs microseconds have passed since at least one input records has been buffered. Defaults to 0.

max_buffering_delay_usecs: u64

Maximal delay in microseconds to wait for min_batch_size_records to get buffered by the controller, defaults to 0.

resources: ResourceConfig

Resource reservations and limits. This is enforced only in Feldera Cloud.

clock_resolution_usecs: Option<u64>

Real-time clock resolution in microseconds.

This parameter controls the execution of queries that use the NOW() function. The output of such queries depends on the real-time clock and can change over time without any external inputs. The pipeline will update the clock value and trigger incremental recomputation at most each clock_resolution_usecs microseconds.

It is set to 1 second (1,000,000 microseconds) by default.

Set to null to disable periodic clock updates.

pin_cpus: Vec<usize>

Optionally, a list of CPU numbers for CPUs to which the pipeline may pin its worker threads. Specify at least twice as many CPU numbers as workers. CPUs are generally numbered starting from 0. The pipeline might not be able to honor CPU pinning requests.

CPU pinning can make pipelines run faster and perform more consistently, as long as different pipelines running on the same machine are pinned to different CPUs.

provisioning_timeout_secs: Option<u64>

Timeout in seconds for the Provisioning phase of the pipeline. Setting this value will override the default of the runner.

max_parallel_connector_init: Option<u64>

The maximum number of connectors initialized in parallel during pipeline startup.

At startup, the pipeline must initialize all of its input and output connectors. Depending on the number and types of connectors, this can take a long time. To accelerate the process, multiple connectors are initialized concurrently. This option controls the maximum number of connectors that can be initialized in parallel.

The default is 10.

init_containers: Option<Value>

Specification of additional (sidecar) containers.

Trait Implementations

impl Clone for RuntimeConfig

fn clone(&self) -> RuntimeConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for RuntimeConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for RuntimeConfig

fn default() -> Self

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for RuntimeConfig

where RuntimeConfig: Default,

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl PartialEq for RuntimeConfig

fn eq(&self, other: &RuntimeConfig) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Serialize for RuntimeConfig

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl<'__s> ToSchema<'__s> for RuntimeConfig

fn schema() -> (&'__s str, RefOr<Schema>)

Return a tuple of name and schema or reference to a schema that can be referenced by the name or inlined directly to responses, request bodies or parameters.

fn aliases() -> Vec<(&'__s str, Schema)>

Optional set of alias schemas for the ToSchema::schema.

impl Eq for RuntimeConfig

impl StructuralPartialEq for RuntimeConfig