Module warehouse 

Writing a job’s results as a medallion warehouse.

A pipeline that produces JSONL produces something a person can read and nothing a query engine can. The warehouse is the same rows in Parquet, laid out the way data engineering already lays this out:

• bronze — every concluded item, success and failure, exactly as the ledger recorded it. Append-only and lossy about nothing. The failures belong here: a bronze layer that silently drops what went wrong is a bronze layer you cannot audit, and “which items failed and why” is a question people ask of the warehouse, not of the log.
• silver — the successful rows, typed against the output type the node declared. Validation is the point of the layer, so a node that declares Ty::Json has nothing to validate against and gets no silver table. That is recorded in the manifest with the reason, rather than emitting one JSON-blob column and calling it typed.
• gold — the rollup node’s own output: curated, aggregate, and by nature defined by whoever wrote the spec rather than by cuttlefish.

Every bronze and silver row carries its own lineage columns. That duplicates data, deliberately: a Parquet file gets copied, attached, and handed to somebody who does not have the manifest, and a row that cannot answer “where did you come from” once separated from its manifest is a row whose provenance depends on filesystem luck.

On the source column

Lineage records the item’s input verbatim, as JSON, in source_input. It would read better to publish a source_uri — but which key of the input holds the path is the spec author’s business, not cuttlefish’s, and a guess (“try path, then url, then file”) produces a column that is right for the corpora we happened to test and silently empty for everyone else. The verbatim input is always correct and always complete.

Structs

Lineage

Lineage carried by every bronze and silver row.

Manifest

The manifest written at the warehouse root.

Row

One concluded item, as the warehouse sees it.

TableEntry

What a manifest says about one table.

Enums

Layer

A layer either has a table or has a reason it doesn’t.

WarehouseError

What went wrong writing a warehouse.

Functions

bronze_batch

Build the bronze batch: every row, success and failure alike.

bronze_schema

The bronze schema: lineage, plus the raw output and error.

entry_for

A TableEntry describing a batch written at path under root.

now_rfc3339

An RFC 3339 timestamp for written_at and for the gold row.

silver_batch

Build the silver batch: successful rows only, typed against item_output.

silver_schema

The columns a node’s declared output type contributes to silver.

write_manifest

Write the manifest to root/manifest.json.

write_parquet

Write one batch to path as Parquet.