Struct MultiFileDataSource 

pub struct MultiFileDataSource { /* private fields */ }

A builder that discovers multiple Vortex files from glob patterns and constructs a MultiLayoutDataSource to scan them as a single data source.

The primary interface is Self::with_glob, which accepts a glob pattern and an optional filesystem. For non-local filesystems (S3, GCS, etc.), callers must provide a FileSystemRef. For local files, pass None and a local filesystem will be created automatically.

Examples

// Local files — filesystem is auto-created:
let ds = MultiFileDataSource::new(session)
    .with_glob("/data/warehouse/*.vortex", None)
    .build()
    .await?;

// S3 — caller provides the filesystem:
let ds = MultiFileDataSource::new(session)
    .with_glob("prefix/*.vortex", Some(s3_fs))
    .build()
    .await?;

// Mixed filesystems — multiple globs with different filesystems:
let ds = MultiFileDataSource::new(session)
    .with_glob("bucket-a/*.vortex", Some(s3_fs.clone()))
    .with_glob("bucket-b/*.vortex", Some(s3_fs))
    .with_glob("gcs-bucket/*.vortex", Some(gcs_fs))
    .build()
    .await?;

Implementations

impl MultiFileDataSource

pub fn new(session: VortexSession) -> Self

Create a new MultiFileDataSource builder.

pub fn with_glob( self, glob: impl Into<String>, fs: Option<FileSystemRef>, ) -> Self

Add a path glob for file discovery.

The glob path should be relative to the filesystem’s base URL. Pass None for the filesystem to use the local filesystem (auto-created in Self::build).

Relative paths are resolved against the process working directory.

pub fn with_open_options( self, f: impl Fn(VortexOpenOptions) -> VortexOpenOptions + Send + Sync + 'static, ) -> Self

Customize VortexOpenOptions applied to each file.

Use this to configure segment caches, metrics registries, or other per-file options.

pub async fn build(self) -> VortexResult<MultiLayoutDataSource>

Build the DataSource.

Discovers files via glob, opens the first file eagerly to determine the schema, and creates lazy factories for the remaining files.