Skip to main content

Crate trading_data

Crate trading_data 

Source
Expand description

Facade/prelude: the derivation engine, indicator state machines, and the storage tier. Depend on this crate only — sub-crates are wiring detail.

Macros§

always_present
The identity Present: an item that is its value, absent only by not being emitted. One line per type because the blanket impl is spent on Option and Rust has no specialization.
graph
Builds a frame from its outputs: the node set, its topological order, the Emitter wrapping, the latches and the Buffers (each sized to the join of every read taken of it) are all derived from the annotated type Deps of the cells the outputs reach.
node_alias
A type alias to a cell, declared so graph can follow it.
slice_nudge
A slice-out cell’s finite-difference witness: copy the batch into Vec<$E> scratch, bump the last element when asked, view it back at the borrow’s own lifetime. Also the cell’s Series declaration — “this out is a run of $E” is exactly what both traits need to know.
value_nudge
A value-out cell’s finite-difference witness: the scratch is just the value itself.

Structs§

Aggregate
A fold over many wire messages: one span per actor, first resetting on every resync.
Armed
An Episodic node’s own contact, sealed in: an ordinary node stepped before N, so N and any leg meant to go dark with it name it in a Gating dep. It folds its trigger at Horizon::Unbounded by construction — a latched bit is exactly the state nothing reconstitutes — which is also why it can never itself be gated.
Arrival
Weaver ordering key. Monotonic by construction, not by measurement: when events arrive faster than the clock resolves, it is the previous key plus one. So it is deliberately not subtractable — the gap between two arrivals is not an elapsed duration.
Asset
Awake
The past that is never read, because the node it would rewind never sleeps. Erased through REWINDS, so a graph driven with this is the sweep it was before anything was anchored.
Bar
Bars
Stateless: both accumulators close a period on the same trade, so the join is this tick’s two batches zipped.
BatchTrades
Batched trade stream event. All trades share prec.
Book
The book fold. Domain-only — it knows nothing of Cell/Node; the graph node wraps this through Book::step, and ShadowBook wraps the same instance, so the persisted stream and the replayed one cannot drift.
BookAnchors
Anchors are Book’s input, not the graph’s — nothing else should name this.
BookChunk
The net effect of one period’s deltas.
BookDelta
One level of our own recollection — see ShadowBook: the persisted delta lane is gapless and self-consistent, so kind (not the venue’s gapped flag) is what a consumer reads. This is the stored row; unlike every other lane holder it carries its own precision, because a Series hands out bare items and a raw price no consumer can scale is not a reading. The persistence tier fills it from the file’s schema.
BookDeltas
BookShape
(price, qty) levels for both sides of an orderbook, keyed by raw price. The wire/persist shape. Both BTreeMaps are ascending; Book holds the same levels best-first.
Buffering
Dep position only, never a frame field: “this series, retained at least R back”. Resolves against the frame’s [Buffer<C, K>] through the Has impl below, whose const-assert proves the declared reach serves the request.
Carried
What a Folds node carries between its elements. Zero at the start of a run and at every reset — a recurrence that needs a different zero says so with a counter slot and a select, which is also how it says it is not warm yet.
Catalog
Close
The kernel of a Closes node.
Dep
A dep as its position: the view a read selects off, and what mints every reading. A body names which element it wants and gets it back tagged with the dep it came from, so provenance records what happened rather than what was claimed.
Elems
A count of elements — Horizon::Elems as a type.
Ema
Nautilus’s ExponentialMovingAverage: α = 2/(len+1), seeded on the first sample, absent until len samples have reached it.
EmaWindow
Ema’s weights, α = 2/(len+1), over the last K elements — see the windowed smoothers above.
Env
What a per-element kernel’s read fills: the body’s env, and — where one is watching — which element of which dep each slot was copied from.
Ex
Operator-ergonomics handle: x*y + constant(2.0) builds the nested type. Delegates every Expr method to its inner primitive.
Exact
Same-actor difference: one clock read twice, so no skew term.
Feather
Typed lane writer: a key/schema mismatch is unrepresentable, push is monomorphic.
Field
A named slot of an item — one per #[slot] field, written by #[derive(Item)]. Carrying the item in the type is what makes a field of some other item a compile error rather than a column silently describing the wrong number.
Fire
One node firing, flattened: values and the finite-difference local Jacobian wrt its deps, à la Jane Street’s “Computations that differentiate and document themselves”.
Fold
The kernel of a Folds node.
Folding
Dep position only, never a frame field: “this dep, H of which I hold myself”. The out is the bare cell’s — nothing wraps the data and nothing is retained for it, so this is a pure declaration, the node’s own recurrence or window stated where it is actually about.
Gains
The up moves of X, and zero where it fell.
Gating
Dep position only, never a frame field: the input that dominates. While it reads false its consumer does not advance — no other dep is pulled, and the out is Latent::latent.
Guide
Constant horizontal guide line in the node’s pane (e.g. RSI 30/70).
Hist
A Buffering dep’s out: all = past ++ fresh, where fresh is byte-identical to the unbuffered series out and past is what stood behind this tick’s batch. horizon is the consumer’s declared one, so a window wider than it stated trips regardless of how far the frame’s buffer happens to reach.
Ink
l/c/a only — hue is the renderer’s; a node can never claim one.
InnerTrade
One trade in raw (scaled-int) form; price/qty are meaningless without the batch’s prec.
LaneReader
Streams one lane’s rows in [start, end], one parquet file at a time. No whole-lane materialization; a mid-stream read failure of a catalog-owned file is unrecoverable and panics.
Lanes
LatencyConfig
A log-normal latency model fitted to three percentiles, seeded deterministically. Build a per-stream LatencySampler with LatencyConfig::sampler: the seed is hashed with the stream’s key, so identical config ⇒ identical stream, distinct keys ⇒ distinct streams.
Live
Live feed. Feed::next drains what’s currently available (blocking for the first event), stamps each message with a monotonic arrival time, weaves one step, and drops consumed rows — memory stays bounded to the un-emitted window regardless of session length. Recording tees incrementally. None once every sink is dropped and the buffer is drained.
LiveClock
Local
Our own process.
Losses
The down moves of X as a positive magnitude, and zero where it rose.
Mc
Market cap. The source reports no event time, so the only reading is ours — this lane’s axis is Local, and venue-named columns would be a lie about whose clock produced it.
McRoot
Moved
A level output as the tick hands it over: the out that stands, and whether this tick moved it.
Ohlc
Ohlcs
The series over a period, in three: the two accumulators over trades, and the bar that is their join. The period is a parameter rather than a name the framework pre-blessed — a node’s identity is still its type, and a Bars over one minute is as distinct from one over five as two newtypes were. Only Tag is new: the period has to reach Cell::NAME for the DAG card to keep saying Bar:1m.
Oi
Open interest, base units.
OiRoot
Over
A window of wall clock — Horizon::Over as a type.
Pair
Part
One named field of one element — Pick’s partial, landing in that field’s own column.
Past
Everything already delivered on the lanes something may be replayed out of, and nothing else. The slices are truncated where the weave stands, so a row it has not emitted is not representable here — which is the whole of what keeps a rewind from reading its own future.
Pending
The period a Closes node is part way through: the accumulator’s slots and the time they close at. Held by the node because the engine owns no storage for one, and read by nothing but the kernel — the body sees the slots as an env and never this struct.
Pick
One element of one dep, at the lag that dep handed it out at — the reading a body puts. Derefs to the element, so a selection may read what is not a slot: an event time above all, which is the kernel’s to carry and never a variable (r[kernels.selection.index-is-not-a-variable]).
Plot
Optional drawing hints a node declares about one group of its Flat slots — the renderer owns everything else (hue above all). Defaults always suffice.
Precision
Decimal exponent: raw = value × 10^precision. Signed, so a million-dollar index tick and a satoshi tick both fit the same 32-bit raw column.
PrecisionPriceQty
Per-batch precision shared across all levels / trades in a book / trade batch.
Predicate
The kernel of a Decides node.
ReadClock
How coarsely a backtest batches, so that it runs faster than the market it replays. That is the whole of it. Arrival time is cut into cells of this length and everything landing in one cell is handed to the graph as a single step, so a replay does a day’s work in as many steps as the cell size buys rather than one per message.
Reading
A number a body may not have: absent exactly where its payload is NaN, which is the representation and not a sentinel a reader has to know about (r[impl outs.absence.typed]). Eight bytes where Option<f64> is sixteen — every bit pattern of an f64 is a valid one, so the discriminant can never be packed into the payload and a retained series of them was paying twice for what the payload already said.
RelayCols
Relay’s columnar twin: per element where the element attests, on the holder where the run does. Both shapes stay — Relay for a single event, this for a run.
Replay
Backtest feed: fills the weaver from the catalog’s lanes, then emits their arrival-ordered merge. Historic rows (no recorded reception) get a simulated arrival via latency.
RotationPolicy
Rows
The default Batch: every row, trimmed on timestamps where its items are Timed and by count where they are not, handed out as a Hist.
Rsi
100 - 100/(1 + U/D) over an average of a series’ Gains and one of its Losses. Which average is the wiring’s to say: Wilder’s is the classic RSI.
RsiDelta
Close-to-close change on B — the series RSI’s up and down moves are taken of.
Sampling
Dep position only: “the last value C produced, whenever that was”. Its slot in the frame is keyed on this type itself — the [carry] behind it is storage, not a cell, the same way an [Emitter] is.
Scan
The kernel of a Scans node: one out element per driving element, each slot body().eval, and the Jacobian body().grad at the last element — exact, and one pass rather than a clone and a re-emit per dep slot.
ShadowBook
We persist our own recollection, not the venue’s story. A gap or a resync is a fact about our connection; stored raw, every replay would re-derive the reconciliation from whatever venue snapshots happened to land. Reconciled once here, replay is exact and free.
Sink
Cloneable push handle: just enqueues. Arrival time is stamped at ingest (single-threaded), so the handle is clock-free and cheap to clone across pump tasks.
Span
The window one actor’s readings of a folded object span. first is the start of the current accumulation epoch — it resets on resync, it is not “when the object was allocated”.
Step
One emission, and the past it stands on. Two values rather than one because Feed::next ties its Lanes to &mut self, so the feed cannot be read alongside the batch it just handed out.
Symbol
Tag
A Cell::NAME a parameter can reach. NAME is a const &'static str, and nothing that formats at runtime can produce one — so the parts are spelled into a fixed buffer, in an associated const, whose value outlives the NAME borrowing it. That an associated const may read its impl’s generics, where a free const item may not, is the whole reason this works.
Tail
A prefix of a run, carrying how far its own end stands behind the run’s newest, and the dep it was cut from. A cross-rate reader cuts a series at its own deadline and then picks inside the cut; without this the pick would report a lag against the cut instead of against the column it lands in.
Timeframe
The period a Bars<TF> is over: naming one is now part of wiring a graph, and a facade-only consumer has no other route to the type. Implemented over the number of milliseconds
Trade
One stored trade, raw as the venue sent it: price/qty are meaningless without the lane’s precision. It is the disk row, never a graph view — nodes read TradeCols.
TradeBuf
Owned twin of TradeCols: lane buffer, Nudge scratch and ingest accumulator in one.
TradeCols
Trades
Ts
Nanoseconds since the Unix epoch, tagged with the actor whose clock produced it.
Unbounded
Reaches to the start of the run — Horizon::Unbounded as a type.
Usd
A struct representing USD (in future, inflation-adjusted) value. That’s it. Just a newtype. But extremely powerful.
Vars
The dep-slot dispenser handed to a trading_data_dag::Symbolic body: v.get::<I>() is the Var reading dep I (bounds enforced at eval time by env length).
Venue
The exchange a fact originates from.
Volume
Volumes
Wilder
Wilder’s running mean: an SMA over the first len samples, then (prev·(len-1) + x)/len. Absent until warm.
WilderWindow
Wilder’s weights, α = 1/len, over the last K elements — see the windowed smoothers above.
Win
A trailing window, every element of which knows its own lag — so a body folding one names no offsets and computes none.
Zipf
Mean of X under weight (k+1)^-s on the sample k back, normalised by the weight the history seen so far carries — which tends to ζ(s).

Enums§

BookUpdate
Distinguishes full snapshots from incremental deltas. For deltas: qty=0 means remove that price level.
CatalogError
Direction
A position’s sign. Flat is a decision — that there is nothing here — where None upstream is the absence of one.
ExchangeName
Fidelity
How much of what a node’s body read the kernel’s fullest reading covers (r[kernels.fidelity.stated]) — Kernel::jac where there is nothing wider, and Kernel::exact where there is. Three answers rather than two, because the middle one is the case nothing else can show: a derivative may be algebraic and still be narrower than the reading it was taken from, and a partial derivative of a body that reads a window looks exactly like an exact one of a body that reads a point.
FrameKind
Whether a level is market activity or our own reconciliation — a UInt8 column exactly parallel to the side byte. Per row, not per run: what a run of levels is, is what its rows say it is, and the one consumer that discriminates (a flow derivation, which a correction would feed fabricated signal) filters on it.
Horizon
How far back a dep position reaches: nothing at all (a bare dep), the engine’s retention (Buffering), or the consumer’s own state (Folding). One vocabulary for all three, so the reach a node reads and the reach it holds are stated the same way — and a const of it drops straight into const-generic position.
Instrument
LaneKind
The source lanes a graph may require. Debug/Hash back the deterministic per-lane latency seed; the router reads only the lanes in the graph’s dep tree. Deltas and anchors are separate kinds: naming one must not load and accumulate the other, since an un-drained lane in Live grows without bound.
Side
TimeframeDesignator
The period a Bars<TF> is over: naming one is now part of wiring a graph, and a facade-only consumer has no other route to the type.
Want
How much of a fire the observer reads. Everything above Want::Vals is second order — an Opaque node’s Jacobian is one clone_from plus one re-advance per scalar dep slot, and a Pure one’s is a grad pass over the pre-advance body — so it is priced per tick, never per build.

Traits§

Batch
How a [Buffer] holds a Series’ retained reach. The default Rows keeps every row; a series whose rows collapse — a book’s levels keep only the last qty per price — implements its own and pays its own state rather than the row count.
Blind
The stated hatch: a node that computes in Rust and therefore has no algebra reading. WHY is not documentation, it is the cost of using it — graph! counts these, and a graph that pins the count only lets it fall.
Bump
Typed-space perturbation of one flattened element, for the finite-difference witness. Returns the perturbation actually applied, in the element’s own units — a quantized column can only move in whole ticks, and pretending otherwise divides the difference by a step never taken. 0.0 ⇒ this slot cannot be perturbed (discrete, structural), so its Jacobian column stays NaN rather than a fabricated zero.
Cell
A value slot in the frame. Out<'t>: Copy — references are Copy, so a batch out enters the frame as &'t [T] and heavy root/node state is lent as &'t State.
Clock
Closes
A run-shaped node whose elements are periods, closed by the first driving element past a boundary. The kernel owns the walk and the close time; the body owns the numbers, as three expressions — what a period’s first element opens with, what a further element folds into it, and what a period no element reached is.
Decides
A scalar bool node: a driving dep that clocks it — the leading one, whose having fired at all is half the verdict — and whatever it reads at their last elements, compared by an Expr.
Episode
Episode lifecycle on an out value; the initial state is the node’s Default.
Episodic
A node that runs a self-terminating episode, latched from inside the graph. Trigger is the one dep that stays live while the episode is dark — the arm — and the node’s own Episode::terminal out drops the contact. Where a hand-written Latch leaves the loop open (nothing checks that the gate you armed is the gate your episode cuts), this closes it in the type: Armed<Self> is the only gate it can be.
Expr
One expression node: evaluate, accumulate exact partials, or project to Ast.
Feed
A source of woven lanes. None = exhausted (end-of-range for Replay; all senders dropped and drained for Live).
Flat
A cell output as a fixed-shape element array: the unit of observation and differentiation. DIMS is the shape ([] scalar, [n] vector, [r, c] row-major, any rank). A batch out (&[T]) flattens to its last element — the observer sees end-of-batch.
Folds
A run-shaped node computing one out element per element of its driving dep, carrying state between them. Two expressions over one env: what the state becomes, and what the element then is.
Gate
A binary control signal. A node naming it through a Gating dep is not advanced while it is false: deps not pulled, no work done, out = Latent::latent. Gates are scalar-out.
Glance
The headline a human reads off a node at a glance — one compact line for the graph viz, the display-dual of Flat’s numeric flattening. A batch renders its last element (empty ⇒ []), matching the observer’s end-of-batch view.
Kernel
How a node computes, and therefore what can be read off it. Sealed: the kernel set is the framework’s, so a node’s only choice is which one it names and there is no way to supply a body the engine cannot also read.
Lagged
A dep out one element of which a reading may select, with how far that element stands behind the run’s newest — the element a dep’s Jac column stands for. Nothing reaches this directly: the element travels as a Pick, which is Dep’s to mint, so no site counts len - 1 - i.
Latch
A Gate armed from outside and cut from within — the SCR/thyristor: an external event (its Deps) sets it, conduction latches in its own state, and it turns off by natural commutation when the episode it gates reaches a Episode::terminal out. No second external signal ever closes it. Cut is read post-sweep; commutation + the gated-node resets are deferred to the next tick’s start (the frame still borrows batch fields at end-of-tick).
Length
A smoothing length — a number an app reading it from a config file has no const to give.
Nudge
A cell’s finite-difference witness: materialize a pulled out into owned scratch (Nudge::stage, bumping one element when bump is Some), then hand it back at a fresh borrow lifetime (Nudge::view). The materialize step owns the data — a slice cell copies its batch into Scratch = Vec<T>, a value cell stores the value in Scratch — which unties the re-advance lifetime from the pulled 't, so re-advanceing a short-lived clone typechecks. No blanket impl — the value/slice cases overlap — so every observed cell writes its own short impl.
Observer
Sees every step_obs as it happens: one interpretation choke point, many interpretations. Step order IS topo order, so the observed sequence doubles as the graph’s static topology; dep names never seen as stepped nodes are roots — apps seed root activations via observe_root.
Present
A Series item read as “did this element carry anything” — what [carry] must ask before it keeps one as a level. The dominant item in this codebase is Option<f64>, a rate-preserving decline; retaining one of those as the standing value would hold an absence forever.
ProbabilisticDistribution
A Flat out whose slots are probabilities over a named outcome space. Evidence reaches one as points rather than shares: a read argues only for what it names, and what nothing argued for is ignorance rather than a uniform guess — so the space has to own a slot to be ignorant in, which is what the Default bound buys.
Put
One reading an Env takes: a copy of one element’s slots, and the three coordinates it came from. Implemented by the tokens a Dep mints and by nothing else — which is what leaves an author no way to state a provenance, and so no way to state a wrong one.
Reach
A reach in dep position, where Horizon is the same thing as a value — what the engine joins and compares. The two vocabularies coexist deliberately: Buffer<C, K> takes a Horizon because graph! computes K as the join of every read of the series, and no type could name a join over reads that have not been seen yet.
Rewound
A past for one node: what it needs to fold itself forward over the ticks it was not stepped on. Indexed by the node and implemented by the driver’s feed — the dag names no storage, so what a past is is not this crate’s to say.
Roots
A graph’s required root events — its dep tree, computed in isolation: the TypeId of each declared root event whose root is consumed by some node. graph! implements it; the facade maps the TypeIds to source lanes.
Row
A lane’s row type. Sealed: the set of lanes is this crate’s contract with the disk.
Runs
The run side’s stated hatch — Blind’s sibling: a node that fills its run in Rust and therefore has no algebra reading. The engine owns the run, so the struct holds only what it remembers between ticks, and emit cannot read what it wrote last tick. &mut self, not &'t mut self: the node is not lent for the tick, only the engine’s buffer is.
Scans
A run-shaped node computing one out element per element of its driving dep — the leading one in Deps, whose rate it therefore preserves — carrying nothing between elements. Every number is an Expr, one per slot of the item; which elements of which deps the expression reads is Rust, licensed by r[kernels.selection.index-is-not-a-variable]: an index is constant wrt every differentiation variable, so it has no slope to state and the body differentiates only values.
Series
A cell whose out is a run of Items — the bufferable shape. The associated Item is what keeps the Buffering Has impl below free of an unconstrained element parameter (E0207). slice_nudge! declares it.
Slots
One expression per slot of a multi-slot out — what a per-element kernel evaluates to fill an item. Heterogeneous, so a tuple rather than Sum’s homogeneous array: the slots of an item are different functions of one env, and a single-slot out is the one-tuple spelled without brackets.
Stamped
An item’s own event time — what an Over reach is located by. A history you cannot index by time is one you can only read at an assumed cadence, which is the bug Horizon::Over replaces.
Symbolic
A scalar-out node whose per-tick value is a pure Expr of its deps, each read at its Flat scalar. Earns the Pure kernel, which is the whole of how it computes: there is no second way to state the value, so the algebra is load-bearing.
Unflat
Flat’s inverse: an item rebuilt from the slots a per-element kernel computed, plus the event time. The time is the kernel’s to carry rather than a slot, because a slot is a differentiation variable and an index may not be one (r[kernels.selection.index-is-not-a-variable]).
Wired
What a cell reads, stated once. Every trait a node is written through — the body traits, Node and Emit — names this as a supertrait rather than declaring a Deps of its own, because a node has one edge set however its body is spelled. Two declarations could disagree, and the Node<Deps = N::Deps> equality bound that used to hold them together was that same statement restated once per kernel.
Witness
Whether a reading is being watched. A type parameter rather than a flag, so the compute path monomorphizes the recording away entirely — kernel_cost’s scan == raw is the check that it did. Sealed: both answers are the engine’s.
ZipfSpec
The exponent of a Zipf — a number an app reading it from a config file has no const to give.

Functions§

abs
absent
How a body inside the algebra declines, which the out plane reads back as absence (r[impl outs.absence.typed]). Every tree reaching one carries Expr::MAYBE, so the operators that cannot answer over one refuse it where it is written rather than where it lands.
closed_by
The prefix of a slower series that has closed by deadline — the cross-rate read a node clocked by a faster series makes against a trading_data_dag::Buffering dep. A Tail and not a slice: the cut hides however many elements stand past the deadline, and a pick inside it still has to report its lag against the column it lands in.
constant
exp
gt
l > r is r < l: one comparison in the algebra, read from the other side.
lt
A comparison is the one thing an absence has no answer for, so a tree that can carry one may not reach here: or it against a number first, or leave the comparison out and let the absence propagate (r[impl outs.absence.typed]).
max
min
or
x where it stands, fallback where it declines. The way out of absence: what comes back is a number exactly as often as fallback is one, which is what lets a body compare it.
powi_of
present
1.0 where x is a number, 0.0 where it declines — how a body reads an absence instead of merely carrying one. Flat, and never absent itself, so it is a condition the comparing operators accept.
read_mc
Bounds are Ts<Local>: market cap has no venue event time, so this lane’s axis is our clock.
read_oi
read_trades
required_lanes
The source lanes a graph requires: maps its Roots::required_events TypeIds to LaneKinds. Lives here — the dag stays storage-free, and persistence can’t see graph types — so it’s the one point that knows both an event’s TypeId and its lane. An unknown event panics.
select
The condition may not decline — a Select is where a body writes an absence, and a branch that cannot be chosen is no branch. The two branches stay permissive, which is the whole point.
sqrt
square
sum

Type Aliases§

DepOuts
Uniform binder-correct dep-tuple type wherever a signature hands a node its pulled deps (concrete types there hit E0195). One alias rather than one per trait: Wired is the only place a Deps is declared, so there is only one projection to name.
DepReads
DepOuts at their positions — what a per-element kernel’s read selects off.
TriggerOut
Binder-correct trigger-out type for Episodic::arms impls — the arm-side DepOuts. Writing the concrete type hits E0195 whenever the trigger’s out carries no lifetime of its own.
UnixNanos

Attribute Macros§

node
Publishes an impl’s type Deps to graph, which cannot ask the type system for it.

Derive Macros§

Item
The out plane’s four mechanical readings of one field list: Flat, Bump, Stamped, and Unflat where the slots carry the whole item.