Struct ConnectorConfig 

pub struct ConnectorConfig {
    pub transport: TransportConfig,
    pub format: Option<FormatConfig>,
    pub index: Option<String>,
    pub output_buffer_config: OutputBufferConfig,
    pub max_batch_size: Option<u64>,
    pub max_worker_batch_size: Option<u64>,
    pub max_queued_records: u64,
    pub paused: bool,
    pub labels: Vec<String>,
    pub start_after: Option<Vec<String>>,
}

A data connector’s configuration

Fields

transport: TransportConfig

Transport endpoint configuration.

format: Option<FormatConfig>

Parser configuration.

index: Option<String>

Name of the index that the connector is attached to.

This property is valid for output connectors only. It is used with data transports and formats that expect output updates in the form of key/value pairs, where the key typically represents a unique id associated with the table or view.

To support such output formats, an output connector can be attached to an index created using the SQL CREATE INDEX statement. An index of a table or view contains the same updates as the table or view itself, indexed by one or more key columns.

See individual connector documentation for details on how they work with indexes.

output_buffer_config: OutputBufferConfig

Output buffer configuration.

max_batch_size: Option<u64>

Maximum number of records from this connector to process in a single batch.

When set, this caps how many records are taken from the connector’s input buffer and pushed through the circuit at once.

This is typically configured lower than max_queued_records to allow the connector time to restart and refill its buffer while a batch is being processed.

Not all input adapters honor this limit.

If this is not set, the batch size is derived from max_worker_batch_size.

max_worker_batch_size: Option<u64>

Maximum number of records processed per batch, per worker thread.

When max_batch_size is not set, this setting is used to cap the number of records that can be taken from the connector’s input buffer and pushed through the circuit at once. The effective batch size is computed as: max_worker_batch_size × workers.

This provides an alternative to max_batch_size that automatically adjusts batch size as the number of worker threads changes to maintain constant amount of work per worker per batch.

Defaults to 10,000 records per worker.

max_queued_records: u64

Backpressure threshold.

Maximal number of records queued by the endpoint before the endpoint is paused by the backpressure mechanism.

For input endpoints, this setting bounds the number of records that have been received from the input transport but haven’t yet been consumed by the circuit since the circuit, since the circuit is still busy processing previous inputs.

For output endpoints, this setting bounds the number of records that have been produced by the circuit but not yet sent via the output transport endpoint nor stored in the output buffer (see enable_output_buffer).

Note that this is not a hard bound: there can be a small delay between the backpressure mechanism is triggered and the endpoint is paused, during which more data may be queued.

The default is 1 million.

paused: bool

Create connector in paused state.

The default is false.

labels: Vec<String>

Arbitrary user-defined text labels associated with the connector.

These labels can be used in conjunction with the start_after property to control the start order of connectors.

start_after: Option<Vec<String>>

Start the connector after all connectors with specified labels.

This property is used to control the start order of connectors. The connector will not start until all connectors with the specified labels have finished processing all inputs.

Implementations

impl ConnectorConfig

pub fn equal_modulo_paused(&self, other: &Self) -> bool

Compare two configs modulo the paused field.

Used to compare checkpointed and current connector configs.

Trait Implementations

impl Clone for ConnectorConfig

fn clone(&self) -> ConnectorConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ConnectorConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for ConnectorConfig

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl PartialEq for ConnectorConfig

fn eq(&self, other: &ConnectorConfig) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Serialize for ConnectorConfig

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl<'__s> ToSchema<'__s> for ConnectorConfig

fn schema() -> (&'__s str, RefOr<Schema>)

Return a tuple of name and schema or reference to a schema that can be referenced by the name or inlined directly to responses, request bodies or parameters.

fn aliases() -> Vec<(&'__s str, Schema)>

Optional set of alias schemas for the ToSchema::schema.

impl Eq for ConnectorConfig

impl StructuralPartialEq for ConnectorConfig