Struct DatasetConfig 

pub struct DatasetConfig {
    pub name: String,
    pub source: SourceConfig,
    pub s3: Option<S3Config>,
    pub index: IndexConfig,
    pub columns: Vec<String>,
    pub dict_encode: bool,
    pub lazy: bool,
}

Fields

name: String

source: SourceConfig

s3: Option<S3Config>

index: IndexConfig

columns: Vec<String>

Optional column projection applied at load time. When non-empty, only the listed columns are read from the parquet/delta source — every other column is skipped entirely (no decode, no allocation, no resident memory). Empty (default) = read all columns. Names are matched case-insensitively against the source schema.

dict_encode: bool

When true (default), Utf8 columns that are dictionary-encoded in the source parquet are read as Arrow Dictionary(Int32, Utf8) instead of being expanded to plain Utf8. Massively cheaper in RAM for low-cardinality columns. Set to false to bypass the override — useful as a workaround if you observe null-handling oddities on a particular parquet file.

lazy: bool

When true, the backend should keep the dataset on disk and stream it at query time instead of materialising it into RAM at startup. Trades the in-memory hot paths (raw Arrow slice, equality index) for bounded memory use on large / multi-file sources. Honoured by the DataFusion backend (local + S3 parquet) and by the DuckDB backend, which registers the dataset as a view over the source scan (local + S3 parquet, and delta) rather than materialising a table.

Implementations

impl DatasetConfig

pub fn resolve_local_parquet_files(&self) -> Result<Vec<PathBuf>, AppError>

Expand source.location to a concrete list of local .parquet files. Only valid for kind = parquet on local paths — S3 and Delta sources are resolved by the backend itself.

Accepts three location shapes:

• a single *.parquet file
• a directory (lists every *.parquet directly inside, non-recursive)
• a glob pattern containing *, ? or […] (e.g. data/year=2024/*.parquet, data/**/*.parquet)

pub fn env_prefix(&self) -> String

Env-var prefix derived from the dataset name: uppercase with non-alphanumeric chars replaced by _. E.g. sales.eu-1 → SALES_EU_1.

pub fn resolved_creds(&self) -> ResolvedCreds

Resolve S3 credentials following the precedence chain documented at the top of this module. Returns an empty struct when nothing was found — the caller should then leave credential resolution to the engine’s default provider chain.

pub fn resolved_region(&self) -> String

Resolved S3 region: per-dataset env (${PREFIX}_AWS_REGION) → inline → AWS_REGION → AWS_DEFAULT_REGION → us-east-1.

Trait Implementations

impl Clone for DatasetConfig

fn clone(&self) -> DatasetConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for DatasetConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for DatasetConfig

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.