Struct ElasticsearchSourceConfig 

pub struct ElasticsearchSourceConfig {
    pub base_url: String,
    pub index: String,
    pub query: Value,
    pub scroll_timeout: String,
    pub auth: AuthSpec<ElasticsearchAuth>,
    pub max_pages: Option<usize>,
    pub batch_size: usize,
}

Available on crate features source-elasticsearch and source-rest only.

Configuration for the Elasticsearch search source.

Fields

base_url: String

Base URL of the Elasticsearch cluster (e.g. "http://localhost:9200").

index: String

Index name to search.

query: Value

Elasticsearch query DSL. Defaults to {"match_all": {}}.

scroll_timeout: String

Scroll context timeout (e.g. "1m"). Defaults to "1m".

auth: AuthSpec<ElasticsearchAuth>

Authentication: either inline ({ type, config }) or a { ref: <name> } pointer to a shared provider in the CLI’s top-level auth: catalog.

max_pages: Option<usize>

Maximum number of scroll pages to fetch. None means no limit.

batch_size: usize

Records per emitted StreamPage, which is also the size parameter passed to the Elasticsearch scroll API (POST /{index}/_search?scroll={timeout}&size={batch_size}). Each scroll response becomes exactly one StreamPage. Defaults to DEFAULT_BATCH_SIZE.

batch_size = 0 is the “no batching” sentinel: the source issues a single non-scroll _search request with size = 10_000 (the default index.max_result_window) and emits one StreamPage. Use it for small indices or for sinks (e.g. SQL COPY, BigQuery load jobs) that prefer one large request to many small ones. Indices that have raised their max_result_window will still cap at 10_000 — raise this knob or switch back to scroll if you need more.

Implementations

impl ElasticsearchSourceConfig

pub fn new( base_url: impl Into<String>, index: impl Into<String>, ) -> ElasticsearchSourceConfig

Create a new config with the required fields and sensible defaults.

pub fn query(self, q: Value) -> ElasticsearchSourceConfig

Set the Elasticsearch query DSL.

pub fn scroll_timeout(self, t: impl Into<String>) -> ElasticsearchSourceConfig

Set the scroll context timeout (e.g. "5m").

pub fn auth(self, a: ElasticsearchAuth) -> ElasticsearchSourceConfig

Set the authentication method.

pub fn max_pages(self, n: usize) -> ElasticsearchSourceConfig

Set the maximum number of scroll pages to fetch.

pub fn with_batch_size(self, batch_size: usize) -> ElasticsearchSourceConfig

Set the per-page document count for both the scroll API’s size parameter and the emitted StreamPage size.

Pass 0 to opt out of scroll entirely — the source will issue a single _search with size = 10_000 and emit one page.

Trait Implementations

impl Clone for ElasticsearchSourceConfig

fn clone(&self) -> ElasticsearchSourceConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ElasticsearchSourceConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for ElasticsearchSourceConfig

fn deserialize<__D>( __deserializer: __D, ) -> Result<ElasticsearchSourceConfig, <__D as Deserializer<'de>>::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl JsonSchema for ElasticsearchSourceConfig

fn schema_name() -> Cow<'static, str>

The name of the generated JSON Schema.

fn schema_id() -> Cow<'static, str>

Returns a string that uniquely identifies the schema produced by this type.

fn json_schema(generator: &mut SchemaGenerator) -> Schema

Generates a JSON Schema for this type.

fn inline_schema() -> bool

Whether JSON Schemas generated for this type should be included directly in parent schemas, rather than being re-used where possible using the $ref keyword.

impl Serialize for ElasticsearchSourceConfig

fn serialize<__S>( &self, __serializer: __S, ) -> Result<<__S as Serializer>::Ok, <__S as Serializer>::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.