Crate datafusion_opensearch 

A DataFusion TableProvider over an OpenSearch (or Elasticsearch) index.

SELECT … FROM <index> WHERE … planned by DataFusion becomes an OpenSearch _search: the WHERE is pushed into the query DSL (pushdown::expr_to_query), the projection into _source, and the LIMIT into the page size. The result is streamed page by page through a scroll cursor (OpenSearchExec), so a scan is complete at any index size with memory bounded by the page size. Filters that cannot be translated exactly are reported TableProviderFilterPushDown::Unsupported, so DataFusion re-applies them in-memory — partial pushdown is therefore always correct and never widens the result set.

Three ways in:

• SQL only: register OpenSearchTableProviderFactory and CREATE EXTERNAL TABLE docs STORED AS OPENSEARCH LOCATION 'http://localhost:9200/my-index'.
• A whole cluster: OpenSearchSchemaProvider lists every index as a table (SHOW TABLES).
• One index: OpenSearchTableFactory::table_provider (schema from the _mapping) or OpenSearchTableProvider::new with a declared Arrow schema.

use datafusion::prelude::SessionContext;
use datafusion_opensearch::OpenSearchTableProviderFactory;

let ctx = SessionContext::new();
OpenSearchTableProviderFactory::new().register(&ctx);
ctx.sql("CREATE EXTERNAL TABLE docs STORED AS OPENSEARCH LOCATION 'http://localhost:9200/my-index'")
    .await?
    .collect()
    .await?;
ctx.sql("SELECT id, status FROM docs WHERE speed > 40 AND status = 'OK' LIMIT 100").await?.show().await?;

With the udf feature, OpenSearch-native predicates — full-text match and the geo bounding box / distance / polygon queries — are available as scalar UDFs (udf) that the provider pushes down as native queries.

Re-exports

pub use catalog::OpenSearchSchemaProvider;

pub use client::OpenSearchClient;

pub use client::SearchPage;

pub use exec::OpenSearchExec;

pub use schema::schema_from_mapping;

pub use table::OpenSearchTableFactory;

pub use table::OpenSearchTableProvider;

pub use table::OpenSearchTableProviderFactory;

pub use table::DEFAULT_PAGE_SIZE;

pub use table::FILE_TYPE;

pub use udf::os_geo_bbox_udf;

pub use udf::os_geo_distance_udf;

pub use udf::os_geo_polygon_udf;

pub use udf::os_match_udf;

pub use udf::OS_GEO_BBOX;

pub use udf::OS_GEO_DISTANCE;

pub use udf::OS_GEO_POLYGON;

pub use udf::OS_MATCH;

Modules

catalog

A SchemaProvider that exposes a cluster’s indices as tables, so SHOW TABLES lists them and SELECT … FROM <schema>.<index> works without registering each index by hand.

client

The HTTP client the provider talks to OpenSearch through.

exec

The physical operator: a _search that streams its result page by page.

pushdown

DataFusion Expr → OpenSearch query-DSL translation.

schema

Arrow schema handling: deriving a schema from an index _mapping, and turning _source documents into a RecordBatch for a (possibly projected) schema.

table

The TableProvider, the factory that builds one from an index’s _mapping, and the TableProviderFactory behind CREATE EXTERNAL TABLE … STORED AS OPENSEARCH.

udf

OpenSearch-native predicates as DataFusion scalar UDFs (feature udf).

Enums

Error

Errors raised while talking to OpenSearch or shaping its responses. Inside a DataFusion plan they surface as DataFusionError::External.

Type Aliases

Result

Convenience alias used throughout the crate.