Crate spate_datagen 

Synthetic commerce-event source for Spate, giving a pipeline you can run with nothing installed.

Every other source in this workspace needs infrastructure before it says anything: a broker, a bucket, a coordination store. That prerequisite is the first thing between a reader and a running pipeline, and it is the only thing this crate removes. Point a pipeline at a DatagenSource and it produces a stream of storefront events (orders, their payments, and refunds against those payments) on as many partitions as you ask for, at a rate you set, for as long as you want.

source:
  datagen:
    partitions: 4
    events_per_tick: 10
    tick_interval: 100ms   # 400 events/s in total
    count: 10000           # omit for an unbounded stream

A dataset, not a schema language

datagen generates one built-in, named dataset. There is no fields: map, and adding one is out of scope rather than unimplemented.

The reason is the thing that makes the stream worth generating. A payment must name an order that was really placed, for an amount that matches its lines, on the same partition, at a later offset. That is a property of the whole dataset, not of any field in it. A field-wise schema can say “a u64 here”; it cannot say “this u64, drawn from the ids the same lane minted earlier and not yet drawn”. A named dataset gets the property for free and stays forty lines of configuration.

Datasets are enumerated by Dataset; the storefront model lives in storefront.

Referential integrity without coordination

Each lane owns a disjoint slice of the order-id space (order_id = n × partitions + lane_index) and keeps its own bounded ring of the orders it has placed and captured. A payment or a refund is drawn from that ring, so it always references an order:

• the same lane minted, and therefore the same partition;
• at a strictly greater offset, because the lane released the order first.

A payment settles the order’s line total exactly. A refund is that total, or its half, third or quarter rounded down, never more than was captured, which is what a balance check downstream rests on.

The mix places faster than it captures, so orders placed and never paid accumulate for as long as the pipeline runs; open_orders reports how many. A check reconciles the orders that were captured, not all of them.

No lane reads another lane’s state, so nothing is shared on the record path and the whole property survives the CPU-pinned fan-out. The payload key carries the order id, so KeyHashRouter colocates an order and its payment in one sink shard.

Delivery, stated plainly

This is a demo and test source. Do not build a production pipeline on it.

• commit() stores watermarks in memory and nowhere else. They are readable through DatagenSource::committed, published as committed_offset when metrics.per_partition_detail is on, and gone when the process exits.
• The source claims no resumability. A restart begins every lane at offset 0, so with a fixed seed the entire stream is regenerated from the beginning, which is strictly more duplication than a real at-least-once source, which would replay only from its last committed position.
• A resume_from: file is declined. A demo source that appears to resume durably gets built on, and the failure surfaces as silent data loss in a deployment that never meant to take one.

Opening the source logs a WARN saying so, once, on the same principle.

Metrics

Families under spate_datagen_source_*: events_generated_total{event}, ticks_total and tick_overrun_total are counted by the lanes; events_remaining, open_orders and (with metrics.per_partition_detail) committed_offset{partition} are published by the control plane.

There is deliberately no spate_source_lag_records. For an unbounded generator the lag is infinite, so the series would exist or not depending on whether count was set.

Modules

storefront

The storefront dataset’s event model, under the name a pipeline assembly reads best:

Structs

DatagenBatch

One poll’s payloads, borrowing the lane’s arena.

DatagenLane

One partition’s data plane.

DatagenSource

Synthetic storefront-event source. See the crate docs for the dataset, the referential-integrity mechanism, and what this source deliberately does not promise.

DatagenSourceConfig

Configuration of a DatagenSource.

OrderLine

One line of an order.

OrderPlaced

A new order, with between one and five lines.

PaymentCaptured

The payment for an order, always preceded in the same partition by the OrderPlaced it names.

RefundIssued

A refund against a payment that was already captured.

Enums

Clock

Where event timestamps come from.

Dataset

Which built-in dataset to generate.

Encoding

Wire format of the generated payloads.

StorefrontEvent

One event in the storefront stream, tagged by type in the encoded payload (order_placed, payment_captured, refund_issued).

Constants

CUSTOMERS

How many distinct customers place orders. Ids run 0..CUSTOMERS.

EVENT_SCHEMA_JSON

The Avro schema of a StorefrontEvent, as JSON.

REGIONS

The regions an order can be placed from.

SKUS

The catalog: 32 stock-keeping units across a handful of product families.