Module schema 

The storage schema: metering types plus what tiering adds.

Column names match metering’s field names deliberately, so the mapping needs no lookup table.

Why the quantity column is value, and carries its own dimension

A field named for kilowatt-hours is true for three of the four Sparten and false for the fourth. Water is metered and billed in m³ (Sparte::billing_unit), and gas registers m³ of Betriebsvolumen before the Brennwert conversion. A column called value_kwh holding a volume is a lie an analyst reads straight past, so the column is col::VALUE and the unit that qualifies it is stored beside it.

That is why col::SPARTE and col::UNIT are core columns rather than deployment-declared extras: a number whose dimension is configuration is a number no query can safely sum.

metering names the field the same way, so the mapping stays a rename-free one (§4.1.1).

Why balancing_day is stored, when derived values are not

This crate refuses to persist anything it can compute: worst_quality is derived on demand precisely so there is no second copy to disagree with the intervals. col::BALANCING_DAY is the one deliberate exception, and the reason is not convenience.

The rule is: the Berlin calendar day for electricity, heat and water, and the Gastag — 06:00 to 06:00 local — for gas. Deriving it needs a zone conversion and a wall-clock (not absolute) six-hour shift, and SQL dialects differ on exactly that, so no single published expression is right everywhere.

Reading the Iceberg files directly is the intended access path, so a rule an external engine cannot express is a rule that will be got wrong — silently, in a daily total that still looks plausible. Storing the answer removes the derivation from every reader instead of publishing four spellings of it and hoping.

The usual objection — a second source of truth that can drift — is answered by there being exactly one writer: to_record_batch_with derives it from metering’s calendar, and nothing else sets it. It also earns its 4 bytes back: a day’s rows share one value, so it dictionary-encodes to nearly nothing, and it is the natural grouping key for every daily aggregate.

Arrow types here are the logical ones. Dictionary encoding is applied by the Parquet writer, not by using Arrow Dictionary types — Parquet’s RLE_DICTIONARY applies to plain string/int columns just as well, and keeping Arrow arrays flat avoids dictionary-unification cost when concatenating batches from different scan chunks.

Modules

col

Column names, in schema order.

Constants

BLOOM_FILTER_COLUMNS

Columns that should carry a Parquet bloom filter.

DELTA_ENCODED_COLUMNS

Columns that benefit from DELTA_BINARY_PACKED: sorted timestamps and the decimal value all store small increments rather than full-width values.

MERGE_KEY

The columns that identify a row for merge resolution.

SORT_COLUMNS

The sort order written into the Parquet footer.

TS_UNIT

Timestamps are microsecond-precision UTC throughout.

VALUE_PRECISION

Decimal precision for value.

VALUE_SCALE

Decimal scale for value. Six places is well beyond metering practice and leaves headroom for gas conversion factors.

VERSION_PRECISION

Precision for version, matching MSCONS’s ≥14-digit numeric label.

VERSION_SCALE

Scale for version — an integer.

Functions

date32

A local date in the Date32 encoding — days since the Unix epoch.

date_of

The inverse of date32, failing rather than panicking on a value out of range — for the reason instant gives.

instant

The inverse of micros.

micros

An instant in the unit the storage schema uses.

storage_schema

Build the storage schema.

timestamp_scalar

A timestamp literal in the exact type the storage schema declares.

timestamp_type

The timestamp type used by all time columns.