Struct KafkaSourceConfig 

pub struct KafkaSourceConfig {

    pub brokers: String,
    pub topics: Vec<String>,
    pub group_id: String,
    pub auth: KafkaAuth,
    pub value_format: KafkaValueFormat,
    pub key_format: Option<KafkaValueFormat>,
    pub auto_offset_reset: OffsetReset,
    pub max_messages: Option<usize>,
    pub idle_timeout: Option<Duration>,
    pub poll_timeout: Duration,
    pub session_timeout: Duration,
    pub on_decode_error: OnDecodeError,
    pub extra_client_config: BTreeMap<String, String>,
    pub batch_size: usize,
}

Available on crate features source-kafka and source-rest only.

Fields

brokers: String

Comma-separated bootstrap server list, e.g. "broker1:9092,broker2:9092".

topics: Vec<String>

One or more topics to subscribe to.

group_id: String

Consumer group ID; required by librdkafka and used for partition assignment.

auth: KafkaAuth

value_format: KafkaValueFormat

key_format: Option<KafkaValueFormat>

Format for the message key. None (the default) decodes key bytes as UTF-8 (or null if there is no key on the message).

auto_offset_reset: OffsetReset

max_messages: Option<usize>

Stop after this many messages have been consumed. At least one of max_messages and idle_timeout must be set.

idle_timeout: Option<Duration>

Stop after this many seconds of no new messages.

poll_timeout: Duration

Single-poll timeout (advisory). Default 1 second.

session_timeout: Duration

Kafka session.timeout.ms. Default 30 seconds.

on_decode_error: OnDecodeError

extra_client_config: BTreeMap<String, String>

Raw librdkafka client properties to pass through. Use with care — these can override anything set by auth or the typed fields above.

batch_size: usize

Messages per emitted StreamPage. Messages drained from the consumer are accumulated into an in-memory buffer and yielded whenever the buffer reaches this size or the idle window flushes a partially-filled buffer. Defaults to DEFAULT_BATCH_SIZE.

batch_size = 0 is the “drain-entire-run-window” sentinel: the source accumulates every message produced by the run (until max_messages or idle_timeout fires) into a single page before yielding. This defeats the point of streaming and exists only for tests or one-shot drain scenarios; prefer a finite batch_size for production pipelines so each batch is durably committed via the state store as soon as the sink confirms the write.

Implementations

impl KafkaSourceConfig

pub fn validate(&self) -> Result<(), FaucetError>

Validate the config at construction time. Called by KafkaSource::new.

pub fn with_batch_size(self, batch_size: usize) -> KafkaSourceConfig

Set the per-page message count for Source::stream_pages.

Pass 0 to drain the entire run window (until max_messages or idle_timeout fires) into a single StreamPage. This is only useful for tests or one-shot drain scenarios — it negates the streaming benefit of incremental sink writes plus per-page bookmark persistence.

Trait Implementations

impl Clone for KafkaSourceConfig

fn clone(&self) -> KafkaSourceConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for KafkaSourceConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for KafkaSourceConfig

fn deserialize<__D>( __deserializer: __D, ) -> Result<KafkaSourceConfig, <__D as Deserializer<'de>>::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl JsonSchema for KafkaSourceConfig

fn schema_name() -> Cow<'static, str>

The name of the generated JSON Schema.

fn schema_id() -> Cow<'static, str>

Returns a string that uniquely identifies the schema produced by this type.

fn json_schema(generator: &mut SchemaGenerator) -> Schema

Generates a JSON Schema for this type.

fn inline_schema() -> bool

Whether JSON Schemas generated for this type should be included directly in parent schemas, rather than being re-used where possible using the $ref keyword.

impl Serialize for KafkaSourceConfig

fn serialize<__S>( &self, __serializer: __S, ) -> Result<<__S as Serializer>::Ok, <__S as Serializer>::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.