Skip to main content

Crate of_persist_parquet

Crate of_persist_parquet 

Source
Expand description

§of_persist_parquet

of_persist_parquet is the opt-in columnar cold-storage companion to of_persist. It exports validated normalized market-data WAL records to Apache Parquet without adding Arrow, compression, or hashing dependencies to the hot persistence crate.

The crate is intentionally a control-plane and research path. Do not call an export method from an adapter polling or execution hot thread.

§Guarantees

  • Hive-style date/venue/symbol/stream partition directories.
  • Stable schema version 1 with sequence, timestamp, quality, source, payload, and optional derived-snapshot columns.
  • Configurable bounded record-batch and row-group sizes.
  • Snappy, Zstandard, or uncompressed output.
  • Temporary-file write followed by atomic, no-clobber same-directory hard-link publication.
  • Optional file synchronization before publication.
  • Full post-write reopen, schema, row-count, sequence-range, and SHA-256 verification before a retention proof is returned.
  • Quality flags are decoded from versioned normalized OFNE records; malformed normalized payloads fail export instead of being silently downgraded.
  • Existing of_persist JSONL, WAL, checkpoint, and retention APIs are unchanged.

§Public API Meaning

MarketDataParquetExportConfig controls a bounded control-plane export, including compression, batch size, row-group size, synchronization, and publication behavior. MarketDataParquetPartitionKey identifies the dataset partition and validates date, venue, symbol, and stream components. MarketDataParquetSourceMetadata records provenance so a cold file can be audited without guessing which adapter or session produced it.

MarketDataParquetWriter consumes selected normalized records from of_persist; it does not own a live WAL or alter the hot persistence API. VerifiedMarketDataParquetExport is returned only after reopen, schema, sequence, row-count, and checksum verification. Its retention input is the proof-bearing handoff that allows a host to consider deleting dependent hot data. Compression and Parquet operations are synchronous and must run on a maintenance worker, never on a feed or execution hot thread.

§Quick Start

use of_persist::{MarketDataWal, MarketDataWalConfig, MarketDataWalReplayFilter};
use of_persist_parquet::{
    MarketDataParquetCompression, MarketDataParquetExportConfig,
    MarketDataParquetPartitionKey, MarketDataParquetSourceMetadata,
    MarketDataParquetWriter,
};

let wal = MarketDataWal::open(MarketDataWalConfig::new("data/normalized.ofmw"))?;
let writer = MarketDataParquetWriter::open(
    MarketDataParquetExportConfig::new("data/cold")
        .with_compression(MarketDataParquetCompression::Zstd)
        .with_batch_rows(32_768)
        .with_row_group_rows(131_072),
)?;
let key = MarketDataParquetPartitionKey::new(
    "2026-08-12", "CME", "ESM6", "market-data",
);
let source = MarketDataParquetSourceMetadata::new(
    "cqg-primary", "cqg-webapi", "session-20260812",
);
let proof = writer.export_wal(
    &key,
    &source,
    &wal,
    MarketDataWalReplayFilter::new(),
    &[],
)?;
assert!(proof.verified);
let retention = proof.retention_input(0, 256 * 1024 * 1024);
assert!(retention.cold_export_verified);

§Partition And Schema Contract

The caller supplies an ISO YYYY-MM-DD partition date. This avoids making UTC session-boundary assumptions inside the library. Components reject empty values, path separators, traversal markers, control characters, and reserved temporary suffixes.

Schema version 1 contains:

ColumnArrow typeMeaning
schema_versionUInt16Export schema version
partition_dateUtf8Caller-selected UTC/session date
venue, symbol, streamUtf8Partition identity
source_id, adapter_id, session_idUtf8Capture provenance
record_kind, record_kind_nameUInt16, Utf8Stable WAL kind
wal_sequenceUInt64Global persisted sequence
provider_sequence, event_sequenceUInt64Source/canonical sequences
ts_exchange_ns, ts_recv_nsUInt64Exchange and receive times
quality_flagsUInt32Effective live quality flags
payloadBinaryOriginal decoded WAL payload
payload_checksumUInt32Per-payload FNV-1a checksum
derived_schema_idnullable UInt32Optional derived snapshot schema
derived_payloadnullable BinaryOptional derived snapshot bytes

Derived snapshots must be strictly ordered by WAL sequence and are joined in a single linear pass. Callers choose their serialization and schema id; this crate does not couple cold export to one analytics snapshot version.

§Verification And Retention

export_records, export_wal, and export_segmented_wal return VerifiedMarketDataParquetExport only after reopening the final file and validating the exact schema, row count, first/last sequence, Parquet metadata, and SHA-256 digest. verify_export can revalidate a proof later.

retention_input(created_ns, hot_bytes) creates an of_persist::MarketDataRetentionInput with cold_export_verified = true. Only a verified proof has this method, preventing an unverified manifest from accidentally authorizing hot-WAL deletion.

§Operational Rules

  • Keep writer options consistent across files in one dataset.
  • Choose batch and row-group bounds from measured memory and query workloads.
  • Publish files to object storage only after local verification succeeds.
  • Reverify the object-store checksum before deleting dependent hot WAL.
  • Keep incident windows and checkpoint dependencies under the conservative of_persist retention planner.
  • Treat source/session identifiers as operational metadata, not credentials.
  • Export is synchronous by design; schedule it on a dedicated maintenance worker or external compactor.
  • Replay bridges materialize the selected WAL range before encoding. Rotate and export sealed, sequence-bounded segments so memory remains operationally bounded for large histories.
  • A failed post-write verification removes the newly published local file. A verified file is never overwritten by a later export of the same range.

§Compatibility

This crate begins at 0.1.0. It depends on the established of_persist 0.4 record model but does not modify it. Applications adopt the columnar path by adding this crate explicitly; existing APIs, serialized JSONL files, native ABI, Python binding, and Java binding remain valid.

Structs§

MarketDataDerivedSnapshotRef
Optional derived analytics snapshot joined to one WAL sequence.
MarketDataParquetExportConfig
Configuration for MarketDataParquetWriter.
MarketDataParquetPartitionKey
Hive-style cold-export partition identity.
MarketDataParquetSourceMetadata
Source provenance repeated in each exported row.
MarketDataParquetWriter
Synchronous verified Parquet cold-export writer.
VerifiedMarketDataParquetExport
Proof produced only after a Parquet file passes full post-write verification.

Enums§

MarketDataParquetCompression
Compression used for a Parquet export.
MarketDataParquetError
Verified Parquet export failure.

Type Aliases§

MarketDataParquetResult
Result type for verified Parquet export operations.