MeterStore

Hot/cold tiered storage for metering time series. PostgreSQL holds the recent interval window at low latency; Apache Iceberg holds the history at analytical scale. A single explicit timestamp separates them, and one SQL statement spans both.

📖 Documentation · API reference · Changelog

Pre-alpha, and unpublished on purpose. Storage, tiering, archival, querying, reproducible reads and completeness work end to end against real PostgreSQL 16 and a real Iceberg warehouse. The API is still settling; integrating against a real workload is what settles it.

The problem

An intelligent measuring system produces one value per measuring point, per OBIS code, per interval. Fifteen minutes is the German settlement grain:

Retention is regulatory — years to decades for the settlement record. PostgreSQL handles the first row comfortably, the second with care, and the third not at all without becoming a full-time job.

But the operational workload genuinely needs Postgres: recent data is written continuously, corrected, and read transactionally by billing and market communication. Meanwhile settlement, forecasting and grid analysis scan years across hundreds of thousands of meters — an object-storage-and-columnar-format problem.

The data has a natural split most systems refuse to exploit: recent intervals are hot and still being corrected; historical intervals are cold and settled. The boundary between them is a timestamp.

How it works

  MeterInterval.from ─────────────────────────────────────▶

  │◀──── Iceberg (cold, settled) ────▶│
                                      │◀── Postgres (hot) ──▶│
  epoch                       tiering_watermark            now

A row's interval start alone decides its tier, so the tiers are disjoint by construction — no deduplication, no merge, no double-counting. Four decisions carry most of the weight:

• The watermark lives inside the Iceberg snapshot. Archival writes the tier boundary into the snapshot summary in the same commit as the data. Iceberg commits are a compare-and-swap, so rows and boundary become durable together or not at all.
• Purge is DROP TABLE, never DELETE. The hot table is time-partitioned, so archiving a window drops exactly one partition. Deleting a day of readings for 100 k meters row by row would leave ~9.6 M dead tuples for autovacuum.
• Corrections are versions, not overwrites. MSCONS corrects a value by versioning it, so the store needs only Iceberg's append — and a past settlement stays reproducible.
• Nothing on the archival path holds a window. Peak memory is the chunk size, not the ~9.6 M-row window.

How tiering works →

Quick start

cargo add meterstore

use meterstore::prelude::*;
use meterstore::hot::PostgresHot;
use std::sync::Arc;
use time::Duration;

let hot = Arc::new(PostgresHot::new(pool));   // a pool you already own

let cold = IcebergSqlCatalog {
    database_url: &db_url,
    warehouse_uri: "s3://bucket/warehouse",   // or file:// memory:// gs:// abfss://
    catalog_name: "meterstore",
    namespace: "metering",
    file_target_bytes: 512 * 1024 * 1024,
    metadata_pool_max_connections: 4,
    auth: &WarehouseAuth { region: Some("eu-central-1".into()), ..Default::default() },
}.build().await?.cold();

let store = MeterStore::builder()
    .hot(hot)
    .cold(cold.clone(), cold.table_provider("readings_versions").await?)
    .table(
        TableConfig::new("readings_versions")
            .settlement_lag(Duration::days(7))
            .archival_step(Duration::DAY)
            .build()?,
    )
    .build()
    .await?;

store.create_tables().await?;

Then write and read:

// Routes each interval to the tier that owns it.
store.append(&[stored_series]).await?;

// One statement, both tiers — with the boundary it was computed against.
let result = store.query(r#"
    SELECT meter_local_day("from") AS day, SUM(value) AS kwh
    FROM readings
    WHERE malo_id = '12345678901'
      AND "from" >= '2025-01-01' AND "from" < '2026-01-01'
    GROUP BY 1 ORDER BY 1
"#).await?;

result.watermark();        // where cold ended and hot began
result.touched_hot_tier(); // whether the answer is only valid for now

readings is version-resolved; readings_versions is the raw audit trail. The naming is load-bearing — see the version-resolution trap before pointing an external engine at the warehouse.

Getting started →

Requirements

The cold tier takes any Arc<dyn Catalog> — SQL, REST, Polaris, Lakekeeper, Glue — and that seam is driven end to end by the test suite rather than asserted. Two are built for you: a PostgreSQL-backed SQL catalogue on the same database as the hot tier, and AWS S3 Tables behind the s3tables feature. Details.

MeterStore needs only SELECT plus ownership of its own tables: no server configuration, no restart, no extension. That is what makes it deployable on RDS, Cloud SQL and Azure Postgres, where an extension-based approach is not.

Relationship to metering

metering    → what a measurement is, and how to compute with it   (zero I/O, no async)
meterstore  → where it lives, how it is tiered, how it is queried  (all I/O)

metering owns intervals, units, quality flags, DST-correct calendars, validation, Ersatzwertbildung, gas conversion and aggregation. MeterStore adds exactly three things: correction versioning, the transaction-time axis, and the tiering boundary.

That boundary is deliberate. Duplicating a domain rule here — a unit conversion, a DST calendar — would create a second implementation to keep correct, and it would drift.

Documentation

Status

Working end to end against real infrastructure: encoding, both tiers, streaming archival, tier-split queries, version resolution with statistics-based elision, reproducible reads on both time axes, completeness, multi-table sessions, routed writes, erasure, schema quarantine, and both serving surfaces.

554 tests — unit, property, and integration against real PostgreSQL 16 and a real Iceberg warehouse, plus an independently implemented correctness oracle over generated workloads. The open-format claim is checked by two foreign engines: DuckDB reads the Parquet and the Iceberg metadata, and PyIceberg — the Iceberg project's own implementation — reads the schema with its field ids, the partition spec, the format version, and the tiering watermark out of the snapshot summary.

Two claims are measured rather than targeted:

• Compression vs PostgreSQL row storage: ~109× (457 B/row against 4.2 B/row), measured over the whole partition tree including indexes. Read it with the caveats in the measurement suite — the fixture's low cardinality flatters it — but the >10× target is met with room to spare.
• Archival memory is bounded by the chunk, not the window, structurally: no stage on the path holds one.

Not yet done: query-latency benchmarks on reference hardware, so the p99 targets remain aspirational; interop against Spark and Trino; a CLI. Compaction and orphan-file cleanup are blocked upstream rather than deferred.

Development

Requires a Rust toolchain and, for integration tests, a running Docker daemon.

just            # list all recipes
just dev        # format + unit tests (no Docker)
just test       # full suite
just check      # everything CI runs
just site       # serve the documentation site

The integration suites are one test binary against one PostgreSQL container. Cargo would otherwise compile each file under tests/ into its own statically linked executable — twenty-odd full copies of DataFusion, Arrow, Iceberg, sqlx and tonic, about 9 GB, which exhausts a CI runner's disk and fails as a linker bus error rather than as "no space left". A container per test cost seven times the wall clock of a database per test, for the same isolation.

datafusion, arrow, iceberg, parquet, metering, time, rust_decimal and sqlx must each appear exactly once in the dependency graph; just deps fails the build otherwise. Two versions of arrow mean two incompatible RecordBatch types, and two of sqlx mean two incompatible PgPool types — neither fails obviously.

License

Dual-licensed under MIT or Apache-2.0, at your option. Part of the mako platform.