ingraph 0.1.0

A declaration-driven indexing framework: records and nodes are written as Rust types, and the storage rows, the dataflow between them and the GraphQL query surface are emitted from the declaration.
Documentation
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
//! The routing capability, the CONSTRUCTION capability beside it, and the DOWNSTREAM SET a
//! one-product node carries.

use super::{
  ItemId, Rings,
  channel::Outbox,
  message::{AckSender, Delivery, Item, Payload},
};

/// WHICH of a node's products have somewhere to go — the connection state, one bit per product in
/// declaration order.
///
/// [`Sink::wanted`]'s answer for a node that emits SEVERAL: a body asks per product, because a
/// product nobody subscribes to is work with no destination while its neighbour's is not. A
/// one-product node reads the same fact through [`any`](Self::any), which is what its `wanted()`
/// always was.
///
/// # The ceiling is 64, and it is stated rather than discovered
///
/// A bitmask, because this is read once per delivery and once per product inside it — an
/// allocation per read would be paid on the hot path for a fact that is a handful of bits. Sixty-
/// four products on one node is far past what a declaration would write, and the derive REFUSES a
/// wider `emits` by name rather than letting a product past the sixty-fourth silently read as
/// unwanted.
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Hash)]
pub struct Wanted(u64);

impl Wanted {
  /// How many products this mask carries — the derive's own ceiling, spelled once.
  pub const CEILING: usize = u64::BITS as usize;

  /// Nothing downstream wants anything.
  #[inline]
  #[must_use]
  pub const fn none() -> Self {
    Self(0)
  }

  /// The ONE-PRODUCT shape: bit zero, set when that product has an edge.
  ///
  /// What [`Sink::connected`]'s default answers, so a downstream set that never heard of products still
  /// fills the mask its consumer reads.
  #[inline]
  #[must_use]
  pub const fn sole(wanted: bool) -> Self {
    Self(wanted as u64)
  }

  /// Also want the product at `index`, counting from zero in declaration order.
  ///
  /// An index at or past [`CEILING`](Self::CEILING) is DROPPED rather than wrapped — a wrap would
  /// make one product's connection state answer for another's, and the derive refuses the
  /// declaration that could reach here.
  #[inline]
  #[must_use]
  pub const fn with(self, index: usize) -> Self {
    if index >= Self::CEILING {
      return self;
    }
    Self(self.0 | (1u64 << index))
  }

  /// Whether the product at `index` has an edge.
  #[inline]
  #[must_use]
  pub const fn at(self, index: usize) -> bool {
    if index >= Self::CEILING {
      return false;
    }
    self.0 & (1u64 << index) != 0
  }

  /// Whether ANY product has an edge — the whole-node question.
  #[inline]
  #[must_use]
  pub const fn any(self) -> bool {
    self.0 != 0
  }
}

/// WHERE a node's outputs go: the capability its out-edges implement, one impl per product.
///
/// # Why it sits on the WIRES and never on the product
///
/// The obvious seat for "how is a `T` routed?" is `T` itself, and it is the wrong one: two nodes
/// that happen to produce the same declared struct would be two `Sink<T>` impls for one `T`, which
/// is `E0119` and unfixable — the app owns the type, so neither impl can be made the other's. A
/// node's WIRES are a different type per node by construction, so the coherence question never
/// arises, and a multi-product node is one downstream set implementing this trait once per product.
///
/// # Backpressure lives HERE, and that is the whole point of the split
///
/// [`deliver`](Self::deliver) AWAITS room on a bounded edge — the driver calls it after the
/// operator has returned, draining the [`Emitter`](crate::Emitter)'s staged buffer into it. The
/// operator's own `emit` is a sync push into that buffer and can never block, so an operator cannot
/// deadlock on its own full output: the wait belongs to the party that owns the loop.
///
/// # The future is `Send`, and the operator's is not
///
/// This is a DRIVER-side capability, like [`AsyncLands`](super::AsyncLands) and [`Ledger`](super::Ledger),
/// and it is awaited inside a body the async engine spawns — so its future crosses threads with the
/// body. An author's `accept` is the opposite case on either face and takes no such bound: a unit's
/// body is the `!Send` home the thread engine exists for.
/// # What the refusal below can and cannot say
///
/// It fires where `{Self}` implements this trait for NOTHING — a type that is not a downstream set,
/// or one whose set carries a different product. It does NOT fire where the impl exists and one of
/// its own bounds is unmet: `rustc` reports the inner bound instead, pointing a `note:` at the impl's
/// `where` clause, so `Downstream<T>`'s `T: Clone` is answered THERE and is documented there rather
/// than restated here. A note that never renders is a note that is not written.
#[diagnostic::on_unimplemented(
  message = "`{Self}` is not a downstream set carrying `{O}`",
  label = "no `Sink<{O}>`",
  note = "a node's out-edges are held by ONE type implementing this trait once per product it \
          carries: `Downstream<{O}>` where the node has a single product, the node's own emitted \
          `<Node>Downstream` where it declared several, or a set written by hand",
  note = "if the type IS a downstream set, the PRODUCT is the mismatch — an edge is a subscription \
          to one product, so a set carrying another one implements this trait at that one and not \
          at `{O}`"
)]
pub trait Sink<O> {
  /// How many edges carry `O`.
  ///
  /// The driver mints one item id PER EDGE — a fan-out joins on all of its arms — so this is the
  /// number it walks, and the index [`deliver`](Self::deliver) names is a position in it.
  fn arity(&self) -> usize;

  /// Whether anything downstream wants `O` at all.
  ///
  /// The connection state, as a read an operator can take before it computes: a product nobody
  /// subscribes to is work with no destination, and the honest answer is to skip it rather than to
  /// emit into a downstream set that will refuse it. Provided, because it is [`arity`](Self::arity)'s
  /// question in the form the [`Emitter`](crate::Emitter) carries.
  #[inline]
  fn wanted(&self) -> bool {
    self.arity() > 0
  }

  /// The connection state PER PRODUCT — what a multi-product body reads before it computes.
  ///
  /// Provided as the one-product answer, which is [`wanted`](Self::wanted) in the mask's own
  /// spelling: a downstream set that carries one product sets bit zero and nothing else. A node that
  /// emits several overrides it, one bit per declared product, and the [`Emitter`](crate::Emitter)
  /// carries the mask into the call.
  #[inline]
  fn connected(&self) -> Wanted {
    Wanted::sole(self.wanted())
  }

  /// Whether edge `edge` carries `output`'s product.
  ///
  /// The ROUTING question, and the reason a multi-product node's outputs do not all go everywhere:
  /// an edge is a subscription to ONE product, so an output takes the edges that carry it and no
  /// others. Provided as *every edge carries it*, which is exactly true of a one-product downstream set —
  /// there the products are all the same one — and is what keeps today's fan-out unchanged.
  ///
  /// An output with no edge carrying it reaches nobody, which is the same fact as an empty wire
  /// set and gets the same answer from the driver: a
  /// [`NoConsumers`](crate::SkipReason::NoConsumers) gap at a pure node, and the ordinary end of a
  /// lane at a landing one.
  #[inline]
  fn carries(&self, edge: usize, output: &O) -> bool {
    let _ = output;
    edge < self.arity()
  }

  /// **WHAT COPIES ONE OUTPUT FOR A SECOND EDGE that carries it** — `None` where this downstream set
  /// gives every output exactly one seat, and so copies none.
  ///
  /// The fan-out MOVES an output into the LAST edge that carries it and copies it for the ones
  /// before, so a copy is spent exactly where one output has MORE THAN ONE seat. That is a fact
  /// about the downstream SET and not about the payload, which is why the capability is asked here
  /// rather than bounded on `O`: a node's own emitted `<Node>Downstream` answers it PER PRODUCT, so
  /// a product with a single subscriber reaches a copy never and demanding [`Clone`] of its payload
  /// would be demanding a capability nothing spends — while its neighbour with three subscribers is
  /// asked for two copies on the same delivery.
  ///
  /// Provided as `None`, which is the ONE-SEAT-PER-PRODUCT answer and is the honest default for a
  /// set that overrides [`carries`](Self::carries) per product. [`Downstream`] — the framework's
  /// N-edges-one-product holder, whose whole purpose IS the fan-out — overrides it, and pays for the
  /// override with a `T: Clone` on its own [`Sink`] impl.
  ///
  /// # A set that promises a seat it cannot fill
  ///
  /// Answering [`carries`](Self::carries) for a second edge while answering `None` here is a
  /// downstream set contradicting itself. The driver neither panics nor drops: the output still
  /// reaches the last edge that carries it, and the ones before are refused BY NAME through the
  /// node's taps — see [`StreamLane`](crate::StreamLane)'s projection.
  #[inline]
  fn duplicate(&self, output: &O) -> Option<O> {
    let _ = output;
    None
  }

  /// A sweep ringer per CONNECTED edge, for a [`Sweeper`](super::Sweeper) that has to wake this
  /// node's CONSUMERS.
  ///
  /// Minted here because only the downstream set holds the sending halves a ringer is cloned from — the
  /// same reason [`Outbox::nudge`] is on the sender rather than on the receiver.
  ///
  /// ERASED to [`Rings`], and that is forced rather than tidy: a multi-product downstream set's edges are
  /// [`Outbox`]es of DIFFERENT types, so no one `Nudge<ItemId, O>` can name them. Nothing is lost —
  /// a ringer carries no payload, the [`Sweeper`](super::Sweeper) already takes
  /// `Vec<Box<dyn Rings>>`, and the type parameter a nudge wore was never read by the party that
  /// rings it.
  fn nudges(&self) -> Vec<Box<dyn Rings>>;

  /// Hand edge `edge` its `batch`, AWAITING room, answerable on `ack`.
  ///
  /// `false` where that consumer has hung up: nothing downstream will ever release these items, and
  /// the driver's honest answer to that is a
  /// [`ServiceInactive`](crate::SkipReason::ServiceInactive) skip rather than a wait nothing ends.
  fn deliver(
    &mut self,
    edge: usize,
    batch: Vec<Item<O>>,
    ack: AckSender<ItemId>,
  ) -> impl Future<Output = bool> + Send;
}

/// **ONE OF A NODE'S OUT-EDGES, AS THE LAUNCH HOLDS IT** — the erased seat a downstream set takes
/// its sending half out of.
///
/// The wiring walk opens a bounded channel per consumer and holds both halves ERASED, because an
/// edge's TYPE is the producer's output and the consumer's input and a graph carries names rather
/// than types. A node's own downstream set is the party that knows the types — one per declared
/// product — so filling one is the walk offering the seat and the SET naming what it takes.
///
/// # TWO DOORS, because minting is only honest when there is ONE product to mint at
///
/// [`open`](Self::open) MINTS the edge where nothing has opened it yet, which is what a one-product
/// set does: there is exactly one type its edge could carry, so opening at it cannot be a guess. A
/// set with SEVERAL products has no such certainty — an edge is a subscription to one of them, and
/// which one is written in the CONSUMER's own declaration, which this walk cannot read — so it takes
/// [`opened`](Self::opened) instead and seats the product the consumer's own inbox already named.
/// Guessing there would open an edge at one product and hand its receiving half to a consumer
/// expecting another.
pub trait Socket {
  /// What this seat refuses with — [`EdgeTypeError`](crate::EdgeTypeError) at the launch floor.
  type Refusal;

  /// The sending half of this edge at `T`, OPENING it where nothing has yet.
  ///
  /// # Errors
  ///
  /// The seat's own refusal where the edge is already open and carries another type — a producer's
  /// output type and its consumer's input type are one type.
  fn open<T>(&mut self) -> Result<Outbox<ItemId, T>, Self::Refusal>
  where
    T: Send + 'static;

  /// The sending half of this edge at `T` where it is ALREADY open — `None` where it carries
  /// another product, and `None` where nothing has opened it at all.
  ///
  /// It opens nothing, which is the whole of its difference from [`open`](Self::open): a set with
  /// several products probes its way to the one this edge carries, and a probe that minted would be
  /// a guess that stuck.
  fn opened<T>(&mut self) -> Option<Outbox<ItemId, T>>
  where
    T: Send + 'static;
}

/// What seating ONE out-edge in a downstream set came to.
///
/// Three answers rather than a `bool`, because the two failures are different mistakes and an
/// operator fixes them differently: a product with two consumers is a TOPOLOGY the set's shape does
/// not have, and an edge no product carries is a walk naming a type the declaration does not have.
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash, derive_more::IsVariant)]
pub enum Attached {
  /// The set took the edge, at the product whose position this is.
  Seated(usize),
  /// The product at this position already has a subscriber and CANNOT COPY for another.
  ///
  /// Not a count: several subscribers to one product is the ordinary topology, and a set serves it
  /// by handing the fan a copy per edge past the first. This is the answer only where that product's
  /// payload has no copy to give — the demultiplexer's decoded buffer or open handle — and it is
  /// what turns a fan the set cannot perform into a named refusal instead of a dropped consumer.
  Occupied(usize),
  /// No product of this set is what the edge carries.
  Foreign,
}

/// **THE CONSTRUCTION HALF OF [`Sink`]** — how a launch fills a downstream set whose products it
/// cannot name.
///
/// [`Sink`] says where a node's outputs GO once the set is built; this says how the set gets built
/// from the erased edges a wiring walk holds. They are separate traits because they are spent by
/// separate parties at separate times — the walk builds, the driver routes — and because the
/// buffered pair's fan-out is `Sink`'s without ever reaching this door.
///
/// # It is what the [`Wires`](crate::Declared::Wires) seat is cashed against
///
/// A graph root's expansion writes one line per field and cannot see how many products the field's
/// declaration emits. `<N as Declared>::Wires<Out>` is the node's own answer, and this trait is what
/// makes that answer SPENDABLE: the walk builds a `Self::default()` and offers it one seat per
/// declared out-edge, and the set decides which product each one is — and whether that product can
/// take another subscriber, which is the one question only the set's own copy doors can answer.
#[diagnostic::on_unimplemented(
  message = "a launch cannot BUILD `{Self}` as a downstream set",
  label = "no `Attaches`",
  note = "this is the seat `Declared::Wires<Out>` names, and a mounted node's set has to be \
          fillable from the erased edges the wiring walk holds: `Downstream<T>` implements it, a \
          node's own emitted `<Node>Downstream` implements it, and a set written by hand states it \
          alongside its `Sink` rows",
  note = "`Sink` says where a built set's outputs GO and this says how the set gets built — a type \
          with the first and not the second can be driven by a hand-wired arrangement and not \
          mounted by a graph root"
)]
pub trait Attaches: Default {
  /// Seat ONE out-edge at whichever of this set's products it carries.
  ///
  /// # Errors
  ///
  /// The seat's own refusal — which only a ONE-product set can earn, because only that road opens
  /// an edge and so only that road can find one already open at another type.
  fn attach<S>(&mut self, socket: &mut S) -> Result<Attached, S::Refusal>
  where
    S: Socket;
}

/// The downstream set of a node with ONE product — however many consumers subscribe to it.
///
/// The framework's own [`Sink`] implementor, and the shape every node in this crate carries today:
/// one product, N edges, each with its own window and its own item ids. A node with SEVERAL
/// products carries a downstream set of its own instead — one `Option<Outbox<…>>` per product, one
/// [`Sink`] impl per product — which is what the node derives emit; this type is that shape with
/// the product count fixed at one.
///
/// Empty is the ordinary state, not an error: a landing node at the end of a lane publishes to
/// nobody, and [`wanted`](Sink::wanted) is what says so.
#[derive(Debug)]
pub struct Downstream<T> {
  edges: Vec<Outbox<ItemId, T>>,
}

impl<T> Downstream<T> {
  /// No edges yet.
  #[inline]
  #[must_use]
  pub const fn new() -> Self {
    Self { edges: Vec::new() }
  }

  /// Carry `outbox` too — one more consumer of the same product.
  #[inline]
  pub fn attach(&mut self, outbox: Outbox<ItemId, T>) -> &mut Self {
    self.edges.push(outbox);
    self
  }

  /// The edges, in the order they were attached — the order [`Sink::deliver`]'s index names.
  #[inline]
  #[must_use]
  pub fn edges(&self) -> &[Outbox<ItemId, T>] {
    &self.edges
  }
}

impl<T> Default for Downstream<T> {
  #[inline]
  fn default() -> Self {
    Self::new()
  }
}

impl<T> FromIterator<Outbox<ItemId, T>> for Downstream<T> {
  #[inline]
  fn from_iter<I: IntoIterator<Item = Outbox<ItemId, T>>>(edges: I) -> Self {
    Self {
      edges: edges.into_iter().collect(),
    }
  }
}

/// ONE product, so every seat it is offered is that product's and the edge is OPENED rather than
/// probed: there is nothing else this set's edges could carry, and a walk that reached a producer
/// before its consumer's own inbox would otherwise have no road at all.
///
/// It never answers [`Attached::Occupied`], and that is the SAME LAW a node's own set keeps rather
/// than an exemption from it. The law is *a product serves as many subscribers as it can copy for*,
/// and this set's own [`Sink`] impl states `T: Clone` — so a payload that reaches this type at all
/// can always copy, and every extra subscriber is simply the next edge. The corner the law refuses
/// is met one stage EARLIER here: a non-copying payload is `E0277` on that `Sink` bound at the
/// mount, naming the same fix the launch-time refusal names.
impl<T> Attaches for Downstream<T>
where
  T: Send + 'static,
{
  #[inline]
  fn attach<S>(&mut self, socket: &mut S) -> Result<Attached, S::Refusal>
  where
    S: Socket,
  {
    let edge = socket.open::<T>()?;
    Self::attach(self, edge);
    // POSITION ZERO always: the answer names the PRODUCT the edge was seated at, and this set has
    // one — its several edges are several consumers of that one product rather than several
    // products.
    Ok(Attached::Seated(0))
  }
}

/// `T: Send` because the future is: what the async engine holds across the await is a flume send of
/// the batch, and a body that crosses threads carries it. Both engines already demand it one level
/// up — a `Resident` is spawned by value on either — so the bound costs a `!Send` product nothing
/// it had.
///
/// `T: 'static` is [`Sink::nudges`]'s: a ringer handed to the [`Sweeper`](super::Sweeper) outlives
/// every borrow by construction — the clock owns it and rings it from its own thread — so the
/// erased box is the `'static` one that party already takes.
///
/// `T: Clone` is [`duplicate`](Sink::duplicate)'s, and it is THIS type's rather than the drivers':
/// one product with N edges is the fan-out itself, so a second consumer here is a second copy of the
/// same value and the payload has to make one. It is asked at the set that spends it rather than at
/// every driver that holds one.
///
/// A payload that does not copy therefore does not reach this type, and the message says `Clone` —
/// the fix being an [`Arc`](std::sync::Arc) or another refcounted carrier where the value is a
/// decoded buffer or an open handle. A node's own emitted `<Node>Downstream` asks the same question
/// PER PRODUCT instead, so one non-copying product costs its copying siblings nothing; what it
/// cannot do is give that product a second subscriber, and the launch says so by name.
impl<T> Sink<T> for Downstream<T>
where
  T: Clone + Send + 'static,
{
  #[inline]
  fn arity(&self) -> usize {
    self.edges.len()
  }

  #[inline]
  fn duplicate(&self, output: &T) -> Option<T> {
    Some(output.clone())
  }

  #[inline]
  fn nudges(&self) -> Vec<Box<dyn Rings>> {
    self
      .edges
      .iter()
      .map(|edge| Box::new(edge.nudge()) as Box<dyn Rings>)
      .collect()
  }

  #[inline]
  async fn deliver(&mut self, edge: usize, batch: Vec<Item<T>>, ack: AckSender<ItemId>) -> bool {
    self.edges[edge]
      .deliver_async(Delivery::new(Payload::Batch(batch), ack))
      .await
      .is_ok()
  }
}

#[cfg(test)]
mod tests;