Struct MultiFileDataSource 

pub struct MultiFileDataSource { /* private fields */ }

A builder that discovers multiple Vortex files from a glob pattern and constructs a MultiLayoutDataSource to scan them as a single data source.

The primary interface is Self::with_glob, which accepts a glob pattern (optionally prefixed with file://). For non-local filesystems (S3, GCS, etc.), callers must also provide a FileSystemRef via Self::with_filesystem).

Examples

// Local files — filesystem is auto-created:
let ds = MultiFileDataSource::new(session)
    .with_glob("/data/warehouse/*.vortex")
    .build()
    .await?;

// S3 — caller provides the filesystem:
let ds = MultiFileDataSource::new(session)
    .with_filesystem(s3_fs)
    .with_glob("prefix/*.vortex")
    .build()
    .await?;

Implementations

impl MultiFileDataSource

pub fn new(session: VortexSession) -> Self

Create a new MultiFileDataSource builder.

pub fn with_glob(self, glob: impl Into<String>) -> Self

Set the path glob for file discovery.

This path should be relative to the filesystem’s base URL.

pub fn with_filesystem(self, fs: FileSystemRef) -> Self

Set the filesystem to use for file discovery and reading.

Required for non-local URLs (S3, GCS, etc.). For file:// or bare path URLs, a local filesystem is created automatically if none is provided.

pub fn with_open_options( self, f: impl Fn(VortexOpenOptions) -> VortexOpenOptions + Send + Sync + 'static, ) -> Self

Customize VortexOpenOptions applied to each file.

Use this to configure segment caches, metrics registries, or other per-file options.

pub async fn build(self) -> VortexResult<impl DataSource>

Build the DataSource.

Discovers files via glob, opens the first file eagerly to determine the schema, and creates lazy factories for the remaining files.