Struct OpenSearchTableProvider 

pub struct OpenSearchTableProvider { /* private fields */ }

A DataFusion table backed by one OpenSearch index.

Implementations

impl OpenSearchTableProvider

pub fn new( base_url: impl Into<String>, index: impl Into<String>, schema: SchemaRef, ) -> Self

base_url e.g. http://localhost:9200; index e.g. my-index; schema declares the columns (and Arrow types) to read from each hit’s _source. Supported column types: Utf8, Int64, Float64, Boolean, Timestamp(Millisecond, _) (see crate::schema::build_batch).

pub fn with_client_and_schema( client: OpenSearchClient, index: impl Into<String>, schema: SchemaRef, ) -> Self

Build over an existing OpenSearchClient (shared connection pool / auth / TLS).

pub fn with_sort(self, sort: Vec<(String, bool)>) -> Self

Push a sort into _search — (field, ascending) pairs. Required for stable pagination (and for a LIMIT to mean “top-N by this order” rather than an arbitrary N).

pub fn with_search_after(self, after: Vec<Value>) -> Self

Continue after a prior page: OpenSearch search_after values, aligned with with_sort. A scan then reads exactly one page (of the LIMIT, or the page size) — the loop is yours.

pub fn with_page_size(self, n: usize) -> Self

Hits per _search page while streaming a result (default DEFAULT_PAGE_SIZE).

pub fn with_max_rows(self, n: Option<usize>) -> Self

Cap the rows a LIMIT-less scan returns. By default a scan streams the whole result.

pub fn with_partitions(self, n: usize) -> Self

Read the index through n DataFusion partitions in parallel, each an OpenSearch sliced scroll (slice: { id, max }). Worthwhile for large scans on multi-shard indices; a query with a LIMIT always uses one partition so the limit is exact. Default 1.

pub fn with_base_filter(self, clauses: Vec<Value>) -> Self

Add query-DSL clauses that are ANDed into the bool.filter of EVERY scan, on top of any pushed-down WHERE. Use this for constraints the caller must never opt out of — e.g. a row-level security predicate, a tenancy filter, or a soft-delete flag. The clauses are opaque OpenSearch DSL (serde_json::Value), so this crate stays agnostic to what they express.

pub fn with_client(self, http: Client) -> Self

Supply a preconfigured reqwest client (timeouts, connection pool, TLS roots, headers).

pub fn index(&self) -> &str

The index this table reads.

pub fn search_body(&self, projected: &SchemaRef, filters: &[Expr]) -> Value

The _search body (without size) a scan with these filters and this projection sends. Exposed so callers can see exactly what was pushed down.

Trait Implementations

impl Debug for OpenSearchTableProvider

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl TableProvider for OpenSearchTableProvider

fn supports_filters_pushdown( &self, filters: &[&Expr], ) -> DataFusionResult<Vec<TableProviderFilterPushDown>>

Report Exact for filters pushdown::expr_to_query can translate (DataFusion drops its Filter node), Unsupported for the rest (DataFusion keeps filtering them). Same predicate as scan uses, so the two never disagree.

fn schema(&self) -> SchemaRef

Get a reference to the schema for this table

fn table_type(&self) -> TableType

Get the type of this table for metadata/catalog purposes.

fn scan<'life0, 'life1, 'life2, 'life3, 'async_trait>( &'life0 self, _state: &'life1 dyn Session, projection: Option<&'life2 Vec<usize>>, filters: &'life3 [Expr], limit: Option<usize>, ) -> Pin<Box<dyn Future<Output = DataFusionResult<Arc<dyn ExecutionPlan>>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait, 'life3: 'async_trait,

Create an ExecutionPlan for scanning the table with optional projection, filter, and limit, described below.

fn constraints(&self) -> Option<&Constraints>

Get a reference to the constraints of the table. Returns:

fn get_table_definition(&self) -> Option<&str>

Get the create statement used to create this table, if available.

fn get_logical_plan(&self) -> Option<Cow<'_, LogicalPlan>>

Get the LogicalPlan of this table, if available.

fn get_column_default(&self, _column: &str) -> Option<&Expr>

Get the default value for a column, if available.

fn scan_with_args<'a, 'life0, 'life1, 'async_trait>( &'life0 self, state: &'life1 dyn Session, args: ScanArgs<'a>, ) -> Pin<Box<dyn Future<Output = Result<ScanResult, DataFusionError>> + Send + 'async_trait>>

where 'a: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, Self: 'async_trait,

Create an ExecutionPlan for scanning the table using structured arguments.

fn statistics(&self) -> Option<Statistics>

Get statistics for this table, if available Although not presently used in mainline DataFusion, this allows implementation specific behavior for downstream repositories, in conjunction with specialized optimizer rules to perform operations such as re-ordering of joins.

fn insert_into<'life0, 'life1, 'async_trait>( &'life0 self, _state: &'life1 dyn Session, _input: Arc<dyn ExecutionPlan>, _insert_op: InsertOp, ) -> Pin<Box<dyn Future<Output = Result<Arc<dyn ExecutionPlan>, DataFusionError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, Self: 'async_trait,

Return an ExecutionPlan to insert data into this table, if supported.

fn delete_from<'life0, 'life1, 'async_trait>( &'life0 self, _state: &'life1 dyn Session, _filters: Vec<Expr>, ) -> Pin<Box<dyn Future<Output = Result<Arc<dyn ExecutionPlan>, DataFusionError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, Self: 'async_trait,

Delete rows matching the filter predicates.

fn update<'life0, 'life1, 'async_trait>( &'life0 self, _state: &'life1 dyn Session, _assignments: Vec<(String, Expr)>, _filters: Vec<Expr>, ) -> Pin<Box<dyn Future<Output = Result<Arc<dyn ExecutionPlan>, DataFusionError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, Self: 'async_trait,

Update rows matching the filter predicates.

fn truncate<'life0, 'life1, 'async_trait>( &'life0 self, _state: &'life1 dyn Session, ) -> Pin<Box<dyn Future<Output = Result<Arc<dyn ExecutionPlan>, DataFusionError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, Self: 'async_trait,

Remove all rows from the table.