MeterStore

Hot/cold tiered storage for metering time series. PostgreSQL holds the recent interval window at low latency; Apache Iceberg holds the history at analytical scale. A single explicit timestamp separates them, and one SQL statement spans both.

📖 Documentation · API reference · Changelog

Pre-alpha, and unpublished on purpose. Storage, tiering, archival, querying, reproducible reads and completeness work end to end against real PostgreSQL 16 and a real Iceberg warehouse. The API is still settling; integrating against a real workload is what settles it.

The problem

An intelligent measuring system produces one value per measuring point, per OBIS code, per interval. Fifteen minutes is the German settlement grain:

Retention is regulatory — years to decades for the settlement record. PostgreSQL handles the first row comfortably, the second with care, and the third not at all without becoming a full-time job.

But the operational workload genuinely needs Postgres: recent data is written continuously, corrected, and read transactionally by billing and market communication. Meanwhile settlement, forecasting and grid analysis scan years across hundreds of thousands of meters — an object-storage-and-columnar-format problem.

The data has a natural split most systems refuse to exploit: recent intervals are hot and still being corrected; historical intervals are cold and settled. The boundary between them is a timestamp.

How it works

  MeterInterval.from ─────────────────────────────────────▶

  │◀──── Iceberg (cold, settled) ────▶│
                                      │◀── Postgres (hot) ──▶│
  epoch                       tiering_watermark            now

A row's interval start alone decides its tier, so the tiers are disjoint by construction — no deduplication, no merge, no double-counting. Four decisions carry most of the weight:

• The watermark lives inside the Iceberg snapshot. Archival writes the tier boundary into the snapshot summary in the same commit as the data. Iceberg commits are a compare-and-swap, so rows and boundary become durable together or not at all.
• Purge is DROP TABLE, never DELETE. The hot table is time-partitioned, so archiving a window drops exactly one partition. Deleting a day of readings for 100 k meters row by row would leave ~9.6 M dead tuples for autovacuum.
• Corrections are versions, not overwrites. MSCONS corrects a value by versioning it, so the store needs only Iceberg's append — and a past settlement stays reproducible.
• Nothing on the archival path holds a window. Peak memory is the chunk size, not the ~9.6 M-row window.

How tiering works →

Quick start

cargo add meterstore

use meterstore::prelude::*;
use meterstore::hot::PostgresHot;
use std::sync::Arc;
use time::Duration;

let hot = Arc::new(PostgresHot::new(pool));   // a pool you already own

let cold = IcebergSqlCatalog {
    database_url: &db_url,
    warehouse_uri: "s3://bucket/warehouse",   // or file:// memory:// gs:// abfss://
    catalog_name: "meterstore",
    namespace: "metering",
    file_target_bytes: 512 * 1024 * 1024,
    metadata_pool_max_connections: 4,
    auth: &WarehouseAuth { region: Some("eu-central-1".into()), ..Default::default() },
}.build().await?.cold();

let store = MeterStore::builder()
    .hot(hot)
    .cold(cold.clone(), cold.table_provider("readings_versions").await?)
    .table(
        TableConfig::new("readings_versions")
            .settlement_lag(Duration::days(7))
            .archival_step(Duration::DAY)
            .build()?,
    )
    .build()
    .await?;

store.create_tables().await?;

Then write and read:

// Routes each interval to the tier that owns it.
store.append(&[stored_series]).await?;

// One statement, both tiers — with the boundary it was computed against.
let result = store.query(r#"
    SELECT meter_local_day("from") AS day, SUM(value) AS kwh
    FROM readings
    WHERE malo_id = '41373559241'
      AND "from" >= '2025-01-01' AND "from" < '2026-01-01'
    GROUP BY 1 ORDER BY 1
"#).await?;

result.watermark();        // where cold ended and hot began
result.touched_hot_tier(); // whether the answer is only valid for now

readings is version-resolved; readings_versions is the raw audit trail. The naming is load-bearing — see the version-resolution trap before pointing an external engine at the warehouse.

meter_local_day is not a convenience, and for gas it is the wrong function. Europe/Berlin observes daylight saving, so the UTC day boundary sits at 01:00 or 02:00 local and grouping on UTC days is wrong every day of the year — but the German gas market does not balance on the calendar day either. A Gastag runs 06:00 to 06:00 local, so a gas Lastgang grouped by the calendar day books six hours a day into the neighbouring Bilanzierungstag, with totals that still look plausible. meter_balancing_day("from", sparte) reads the commodity per row and picks the right one:

SELECT sparte, meter_balancing_day("from", sparte) AS day, SUM(value)
FROM readings GROUP BY 1, 2;

The DST anomaly moves with the boundary: the clocks change before 06:00, so the 25-hour gas day is the one named after the Saturday while the 25-hour calendar day is the Sunday.

An external engine does not get that function — so it gets the answer instead. SQL dialects differ on timestamp arithmetic, so no single published expression is right everywhere. The encoder applies the calendar once, at write time, and stores the answer:

-- Every engine. No zone conversion, no DST reasoning, no dialect.
SELECT balancing_day, SUM(value) FROM readings GROUP BY 1;

Getting started →

Requirements

The cold tier takes any Arc<dyn Catalog> — SQL, REST, Polaris, Lakekeeper, Glue — and that seam is driven end to end by the test suite rather than asserted. Two are built for you: a PostgreSQL-backed SQL catalogue on the same database as the hot tier, and AWS S3 Tables behind the s3tables feature. Details.

MeterStore needs only SELECT plus ownership of its own tables: no server configuration, no restart, no extension. That is what makes it deployable on RDS, Cloud SQL and Azure Postgres, where an extension-based approach is not.

Relationship to metering

metering    → what a measurement is, and how to compute with it   (zero I/O, no async)
meterstore  → where it lives, how it is tiered, how it is queried  (all I/O)

metering owns intervals, units, quality flags, DST-correct calendars, validation, Ersatzwertbildung, gas conversion and aggregation. MeterStore adds exactly three things: correction versioning, the transaction-time axis, and the tiering boundary.

That boundary is deliberate. Duplicating a domain rule here — a unit conversion, a DST calendar — would create a second implementation to keep correct, and it would drift.

Documentation

Status

Working end to end against real infrastructure: encoding, both tiers, streaming archival, tier-split queries, version resolution with statistics-based elision, reproducible reads on both time axes, completeness, multi-table sessions, routed writes, erasure, schema quarantine, and both serving surfaces.

587 tests — unit, property, and integration against real PostgreSQL 16 and a real Iceberg warehouse, plus an independently implemented correctness oracle over generated workloads. DuckDB and PyIceberg read the output and agree with it.

Measured rather than targeted: ~109× compression against PostgreSQL row storage (457 B/row against 4.2 B/row — the measurement suite carries the caveats), and archival memory bounded by the chunk, not the window.

Not yet done: query-latency benchmarks on reference hardware, so the p99 targets remain aspirational; Spark and Trino interop; a CLI. Compaction and orphan-file cleanup are blocked upstream rather than deferred.

Development

Requires a Rust toolchain and, for integration tests, a running Docker daemon.

just            # list all recipes
just dev        # format + unit tests (no Docker)
just test       # full suite
just check      # everything CI runs
just site       # serve the documentation site

The integration suites are one test binary against one PostgreSQL container. Cargo would otherwise compile each file under tests/ into its own statically linked executable — twenty-odd full copies of DataFusion, Arrow, Iceberg, sqlx and tonic, about 9 GB, which exhausts a CI runner's disk and fails as a linker bus error rather than as "no space left". A container per test cost seven times the wall clock of a database per test, for the same isolation.

datafusion, arrow, iceberg, parquet, metering, time, rust_decimal and sqlx must each appear exactly once in the dependency graph; just deps fails the build otherwise. Two versions of arrow mean two incompatible RecordBatch types, and two of sqlx mean two incompatible PgPool types — neither fails obviously.

License

Dual-licensed under MIT or Apache-2.0, at your option. Part of the mako platform.