1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
//! Synthetic commerce-event source for Spate, giving a pipeline you can run
//! with nothing installed.
//!
//! Every other source in this workspace needs infrastructure before it says
//! anything: a broker, a bucket, a coordination store. That prerequisite is
//! the first thing between a reader and a running pipeline, and it is the only
//! thing this crate removes. Point a pipeline at a [`DatagenSource`] and it
//! produces a stream of storefront events (orders, their payments, and
//! refunds against those payments) on as many partitions as you ask for, at a
//! rate you set, for as long as you want.
//!
//! ```yaml
//! source:
//! datagen:
//! partitions: 4
//! events_per_tick: 10
//! tick_interval: 100ms # 400 events/s in total
//! count: 10000 # omit for an unbounded stream
//! ```
//!
//! # A dataset, not a schema language
//!
//! `datagen` generates **one built-in, named dataset**. There is no `fields:`
//! map, and adding one is out of scope rather than unimplemented.
//!
//! The reason is the thing that makes the stream worth generating. A payment
//! must name an order that was really placed, for an amount that matches its
//! lines, on the same partition, at a later offset. That is a property
//! of the whole dataset, not of any field in it. A field-wise schema can say
//! "a `u64` here"; it cannot say "this `u64`, drawn from the ids the same lane
//! minted earlier and not yet drawn". A named dataset gets the property for
//! free and stays forty lines of configuration.
//!
//! Datasets are enumerated by [`Dataset`]; the storefront model lives in
//! [`storefront`].
//!
//! # Referential integrity without coordination
//!
//! Each lane owns a **disjoint** slice of the order-id space
//! (`order_id = n × partitions + lane_index`) and keeps its own bounded ring
//! of the orders it has placed and captured. A payment or a refund is drawn
//! from that ring, so it always references an order:
//!
//! - the same lane minted, and therefore the **same partition**;
//! - at a **strictly greater offset**, because the lane released the order
//! first.
//!
//! A payment settles the order's line total exactly. A refund is that total,
//! or its half, third or quarter rounded down, never more than was captured,
//! which is what a balance check downstream rests on.
//!
//! The mix places faster than it captures, so orders placed and never paid
//! accumulate for as long as the pipeline runs; `open_orders` reports how
//! many. A check reconciles the orders that were captured, not all of them.
//!
//! No lane reads another lane's state, so nothing is shared on the record
//! path and the whole property survives the CPU-pinned fan-out. The payload
//! key carries the order id, so
//! [`KeyHashRouter`](spate_core::sink::KeyHashRouter) colocates an order and
//! its payment in one sink shard.
//!
//! # Delivery, stated plainly
//!
//! **This is a demo and test source. Do not build a production pipeline on
//! it.**
//!
//! - `commit()` stores watermarks **in memory and nowhere else**. They are
//! readable through [`DatagenSource::committed`], published as
//! `committed_offset` when `metrics.per_partition_detail` is on, and gone
//! when the process exits.
//! - The source claims **no resumability**. A restart begins every lane at
//! offset 0, so with a fixed seed the entire stream is regenerated from the
//! beginning, which is strictly *more* duplication than a real
//! at-least-once source, which would replay only from its last committed
//! position.
//! - A `resume_from:` file is **declined**. A demo source that appears to
//! resume durably gets built on, and the failure surfaces as silent data
//! loss in a deployment that never meant to take one.
//!
//! Opening the source logs a `WARN` saying so, once, on the same principle.
//!
//! # Metrics
//!
//! Families under `spate_datagen_source_*`: `events_generated_total{event}`,
//! `ticks_total` and `tick_overrun_total` are counted by the lanes;
//! `events_remaining`, `open_orders` and (with `metrics.per_partition_detail`)
//! `committed_offset{partition}` are published by the control plane.
//!
//! There is deliberately no `spate_source_lag_records`. For an unbounded
//! generator the lag is infinite, so the series would exist or not depending
//! on whether `count` was set.
pub use ;
pub use ;
pub use ;
pub use ;
pub use DatagenSource;
/// The storefront dataset's event model, under the name a pipeline assembly
/// reads best:
///
/// ```
/// use spate_datagen::storefront::{OrderLine, OrderPlaced, PaymentCaptured, RefundIssued};
/// ```
///
/// The same types are re-exported at the crate root; this module is the
/// spelling to prefer when a file also imports the source and its
/// configuration, because it says which dataset the events belong to.