Struct ConnectorConfig

pub struct ConnectorConfig {
    pub transport: TransportConfig,
    pub format: Option<FormatConfig>,
    pub index: Option<String>,
    pub output_buffer_config: OutputBufferConfig,
    pub max_batch_size: u64,
    pub max_queued_records: u64,
    pub paused: bool,
    pub labels: Vec<String>,
    pub start_after: Option<Vec<String>>,
}

A data connector’s configuration

Fields

transport: TransportConfig

Transport endpoint configuration.

format: Option<FormatConfig>

Parser configuration.

index: Option<String>

Name of the index that the connector is attached to.

This property is valid for output connectors only. It is used with data transports and formats that expect output updates in the form of key/value pairs, where the key typically represents a unique id associated with the table or view.

To support such output formats, an output connector can be attached to an index created using the SQL CREATE INDEX statement. An index of a table or view contains the same updates as the table or view itself, indexed by one or more key columns.

See individual connector documentation for details on how they work with indexes.

output_buffer_config: OutputBufferConfig

Output buffer configuration.

max_batch_size: u64

Maximum batch size, in records.

This is the maximum number of records to process in one batch through the circuit. The time and space cost of processing a batch is asymptotically superlinear in the size of the batch, but very small batches are less efficient due to constant factors.

This should usually be less than max_queued_records, to give the connector a round-trip time to restart and refill the buffer while batches are being processed.

Some input adapters might not honor this setting.

The default is 10,000.

max_queued_records: u64

Backpressure threshold.

Maximal number of records queued by the endpoint before the endpoint is paused by the backpressure mechanism.

For input endpoints, this setting bounds the number of records that have been received from the input transport but haven’t yet been consumed by the circuit since the circuit, since the circuit is still busy processing previous inputs.

For output endpoints, this setting bounds the number of records that have been produced by the circuit but not yet sent via the output transport endpoint nor stored in the output buffer (see enable_output_buffer).

Note that this is not a hard bound: there can be a small delay between the backpressure mechanism is triggered and the endpoint is paused, during which more data may be queued.

The default is 1 million.

paused: bool

Create connector in paused state.

The default is false.

labels: Vec<String>

Arbitrary user-defined text labels associated with the connector.

These labels can be used in conjunction with the start_after property to control the start order of connectors.

start_after: Option<Vec<String>>

Start the connector after all connectors with specified labels.

This property is used to control the start order of connectors. The connector will not start until all connectors with the specified labels have finished processing all inputs.

Trait Implementations

impl Clone for ConnectorConfig

fn clone(&self) -> ConnectorConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ConnectorConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for ConnectorConfig

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl PartialEq for ConnectorConfig

fn eq(&self, other: &ConnectorConfig) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Serialize for ConnectorConfig

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl<'__s> ToSchema<'__s> for ConnectorConfig

fn schema() -> (&'__s str, RefOr<Schema>)

Return a tuple of name and schema or reference to a schema that can be referenced by the name or inlined directly to responses, request bodies or parameters.

fn aliases() -> Vec<(&'__s str, Schema)>

Optional set of alias schemas for the ToSchema::schema.

impl Eq for ConnectorConfig

impl StructuralPartialEq for ConnectorConfig