ingraph 0.1.0

A declaration-driven indexing framework: records and nodes are written as Rust types, and the storage rows, the dataflow between them and the GraphQL query surface are emitted from the declaration.
Documentation
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
//! The PACKAGE a query renders into — [`MongoPipeline`] — and the three assemblies that build one.
//!
//! # It is a value, and that is the whole of the two-layer split
//!
//! A SQL lane renders into a `sqlx::QueryBuilder`, which holds the driver's own argument buffer and
//! therefore cannot exist without the driver. A MongoDB pipeline is a list of BSON documents, which
//! is a value the wire format defines and no client library owns — so the rendering layer hands one
//! back and a caller may run it themselves, hand it to a shell, log it, or pass it to the lane that
//! runs it. That is what makes the lower layer a public escape hatch rather than an internal step.
//!
//! # The stage ORDER is fixed, and it is the only order that answers
//!
//! `$match`, `$sort`, `$skip`, `$limit` — every lane writes a prefix of that sequence and nothing
//! reorders it. A pipeline is applied in the order it is written, so `$limit` before `$sort` sorts
//! an arbitrary N documents and `$skip` after `$limit` skips inside the page instead of before it.
//! Each of those renders without an error and answers a different question, which is why the
//! sequence is written once here rather than assembled per lane.
//!
//! # The `$match` is ONE stage, filter and keyset together
//!
//! Both go through the same [`MongoConjunction`], so a page is *the client's filter AND the resume
//! predicate* and neither conjunct has to know what the other was — the arrangement
//! `query::sql`'s `predicates` already makes, with the accumulator doing the joining. Two `$match`
//! stages would be the same predicate with an extra pipeline boundary the planner has to fuse back
//! together.
//!
//! # A RELATION adds stages between the `$match` and the `$sort`, and the reads grow with them
//!
//! `$lookup` is a stage rather than a key, so a filter carrying a relation finishes as a `$match`
//! AND a run of join stages — see [`MongoPredicate`], which is what the accumulator hands back. They
//! land after the `$match` because conjuncts commute and filtering on the document's own fields
//! first is a join over fewer documents; they land before the `$sort` because a document a join
//! excludes must not be ordered, skipped or counted toward the page.
//!
//! Each join names the collection it reads, and those join [`MongoPipeline::reads`] beside the one
//! the pipeline runs over — which is what flips the execution lane's read concern to a snapshot. So
//! the policy is derived from what the query actually joined rather than from what the declaration
//! might have.
//!
//! # A NESTED ORDER puts its stages in FRONT of all of them
//!
//! An ordering hop is a join too — see [`subject`](super::subject) — and its stages run BEFORE the
//! `$match` where a relation's run after it. The asymmetry is one fact: the resume predicate
//! compares the value those stages materialize, and there is nothing to compare until `$addFields`
//! has written it. A relation's join is a conjunct and conjuncts commute, so it may wait until the
//! `$match` has thinned the input; an order's cannot.
//!
//! The ONE-`$match` ruling above is what gives the position rather than the count: splitting the
//! filter out in front of the ordering joins would make the `$match` two stages free to disagree
//! about which document they were addressed at, so the stage stays one and moves.

use bson::{Bson, Document};

use super::{
  CrossingError, MongoFields, MongoKeyset, MongoOrder, MongoQuery, MongoQueryError, MongoSubjects,
  error::offset, subject::joins,
};
use crate::{
  ComplexComparison, FieldName, Identifiable, Record, Walk,
  comparison::{
    At,
    mongo::{
      MATCH, Mongo, MongoCarrier, MongoConjunction, MongoOperator, MongoPredicate, field_key,
    },
  },
};

/// The `$sort` stage's key.
const SORT: &str = "$sort";

/// The `$skip` stage's key.
const SKIP: &str = "$skip";

/// The `$limit` stage's key.
const LIMIT: &str = "$limit";

/// The predicate a filter over `Row` crosses into, read off the RECORD's own vocabulary binding.
///
/// One name for a two-step projection, for the reason
/// `query::sql`'s own `RowComparison` alias exists: it is spent in several signatures, and what it
/// says is that a comparison is a fact about a record in a database rather than a slot of the
/// engine's.
type RowComparison<Row> = <<Row as MongoQuery>::Record as Record<Mongo>>::Comparison;

/// One query, packaged: which collection it runs against, the stages it runs, and every collection
/// those stages READ.
///
/// # `reads` is a FACT the package carries, not a hint
///
/// It is what the execution lane's read concern is chosen from — a pipeline touching more than one
/// collection needs a snapshot to see one state of the database, and one touching a single
/// collection does not. Carrying it here rather than deriving it at execution is what keeps the
/// policy honest: the party that ADDED a `$lookup` is the party that names the collection it reads,
/// so a stage cannot be added without the fact moving with it.
///
/// For a flat query it holds exactly [`collection`](Self::collection). It is a list rather than an
/// `Option<extra>` because a pipeline reading three collections is the ordinary `$lookup` case, and
/// because the read concern's question — *is this more than one* — is a length.
///
/// # Each NAME is `&'static` and the SET is not, which is what a join changes
///
/// A collection name is declared with the schema and is an ITEM, exactly as a
/// [`RecordAddress`](crate::RecordAddress)'s table is. Which of them one query reads is not: a
/// filter decides whether it joins at all and how deep, so the set is assembled per query beside the
/// stages that made it. [`MongoQuery::READS`] is where the declaration's own baseline is stated, and
/// this list is that plus whatever the filter joined.
#[derive(Debug, Clone, PartialEq)]
#[must_use = "a package renders nothing until it is run"]
pub struct MongoPipeline {
  collection: &'static FieldName,
  pipeline: Vec<Document>,
  reads: Vec<&'static FieldName>,
}

impl MongoPipeline {
  /// A package over `collection`, running `pipeline`, reading `reads`.
  ///
  /// Public because the escape hatch is: a caller assembling a pipeline this crate cannot yet
  /// render — a `$unionWith`, a search stage, a `$graphLookup` — packages it here and runs it
  /// through the same lane, and the read-concern policy applies to theirs exactly as it does to this
  /// crate's.
  #[inline]
  pub const fn new(
    collection: &'static FieldName,
    pipeline: Vec<Document>,
    reads: Vec<&'static FieldName>,
  ) -> Self {
    Self {
      collection,
      pipeline,
      reads,
    }
  }

  /// The collection this pipeline runs against.
  #[inline]
  #[must_use]
  pub const fn collection(&self) -> &'static FieldName {
    self.collection
  }

  /// The stages, in the order they run.
  #[inline]
  #[must_use]
  pub fn pipeline(&self) -> &[Document] {
    &self.pipeline
  }

  /// Every collection these stages read — the main one, and whatever a join added.
  #[inline]
  #[must_use]
  pub const fn reads(&self) -> &[&'static FieldName] {
    self.reads.as_slice()
  }

  /// Whether this pipeline reads MORE THAN ONE collection, which is what the execution lane's
  /// default read concern is chosen on.
  ///
  /// A method rather than a `reads().len() > 1` at the call site, because the POLICY is one
  /// sentence and a comparison written at each caller is a policy free to differ between them.
  #[inline]
  #[must_use]
  pub const fn spans_collections(&self) -> bool {
    self.reads.len() > 1
  }

  /// The stages, taken — what the driver's `aggregate` consumes.
  #[inline]
  #[must_use]
  pub fn into_pipeline(self) -> Vec<Document> {
    self.pipeline
  }
}

/// The stage list, assembled in the one order that answers.
///
/// A `$match` is written ALWAYS, empty filter included: `{$match: {}}` matches every document, and
/// writing it unconditionally is what makes every package one shape a reader can check. The join
/// stages follow it and the other three follow them, because each of the three CHANGES the answer —
/// a `$limit` nobody asked for is a truncated result, a `$sort` nobody asked for is work every
/// document pays for, and either applied before a join has excluded a document is a page assembled
/// out of rows the filter refuses.
fn stages(
  predicate: MongoPredicate,
  sort: Option<Document>,
  skip: Option<i64>,
  limit: Option<i64>,
) -> (Vec<Document>, Vec<&'static FieldName>) {
  let (filter, joins, reads) = predicate.into_parts();
  let mut stages = Vec::with_capacity(joins.len() + 4);
  stages.push(stage(MATCH, Bson::Document(filter)));
  stages.extend(joins);
  if let Some(sort) = sort {
    stages.push(stage(SORT, Bson::Document(sort)));
  }
  if let Some(skip) = skip {
    stages.push(stage(SKIP, Bson::Int64(skip)));
  }
  if let Some(limit) = limit {
    stages.push(stage(LIMIT, Bson::Int64(limit)));
  }
  (stages, reads)
}

/// Every collection one package reads: the declaration's own baseline, and whatever the filter
/// joined, without repeats.
///
/// The baseline is [`MongoQuery::READS`] rather than the address's table alone, so a document type
/// that names further collections of its own keeps them — the default is the one table and a binding
/// may say more.
fn reads<Row: MongoQuery>(joined: Vec<&'static FieldName>) -> Vec<&'static FieldName> {
  let mut reads = Row::READS.to_vec();
  for collection in joined {
    if !reads.contains(&collection) {
      reads.push(collection);
    }
  }
  reads
}

/// `{name: value}` — one stage, which is a document with exactly one key.
///
/// Reached by [`subject`](super::subject) too, so a nested order's `$addFields` and `$project` and
/// the four above are one spelling of what a stage IS.
pub(super) fn stage(name: &str, value: Bson) -> Document {
  let mut stage = Document::new();
  stage.insert(name, value);
  stage
}

/// The `$match` document `filter` and `extra` together make — and whatever a relation joined beside
/// it — threaded through ONE accumulator.
///
/// `K` is a second comparison rather than a keyset by name, because the two callers pass different
/// ones: a page passes its keyset and `get` passes nothing at all, holding the id predicate in the
/// document directly. Both are conjuncts beside the filter, which is the whole of what this needs to
/// know about either.
///
/// It finishes through [`MongoConjunction::into_predicate`] rather than `into_document`, which is
/// the difference a join makes: a `$lookup` is a stage and a filter document has no position for
/// one, so the finisher that answers with the document alone would drop it.
fn predicate<Row, K>(
  filter: Option<&RowComparison<Row>>,
  extra: Option<&K>,
) -> Result<MongoPredicate, MongoQueryError>
where
  Row: MongoQuery,
  K: ComplexComparison<Mongo>,
{
  let address = Row::ADDRESS;
  let at = At::new(&address);
  let mut conjunction = MongoConjunction::new();
  if let Some(filter) = filter {
    conjunction = filter.apply(at, conjunction)?;
  }
  if let Some(extra) = extra {
    conjunction = extra.apply(at, conjunction)?;
  }
  Ok(conjunction.into_predicate())
}

/// `{_id: {$eq: …}}` — the FILTER document that names one stored document, by its id.
///
/// # One route, because the read and the write both take it
///
/// [`address`] wraps it in a `$match` and the write lane's removal hands it to `delete_one` and to
/// `replace_one` — three positions, one spelling. A second copy would be the one place the document a
/// `get` reads and the document a `delete` removes could come to disagree about what an address is.
///
/// The key goes through [`field_key`], which is the one place the record's own id becomes the
/// document's `_id` — see [`write_field_path`](crate::comparison::mongo::write_field_path)'s note
/// for why that mapping has exactly one home.
///
/// # Errors
///
/// Whatever the declaration's own id conversion refuses with — see [`MongoQuery::id`].
pub(super) fn identity<Row: MongoQuery>(
  id: &<Row::Record as Identifiable>::Id,
) -> Result<Document, CrossingError> {
  let address = Row::ADDRESS;
  let mut filter = Document::new();
  let mut operators = Document::new();
  operators.insert(MongoOperator::Eq.as_str(), Row::id(id)?);
  filter.insert(
    field_key(&address, address.keys()[0]),
    Bson::Document(operators),
  );
  Ok(filter)
}

/// The pipeline that names ONE document, by its id.
///
/// No `$limit`, and that is PROVABLE rather than asserted: `_id` is unique by the server's own
/// construction — MongoDB indexes it that way for every collection and refuses a duplicate — so the
/// predicate matches at most one document by the storage engine's law. A `$limit: 1` would restate
/// that law in the pipeline, and would hide a duplicate id rather than letting the index report it.
///
/// # Errors
///
/// Whatever the declaration's own id conversion refuses with — see
/// [`MongoQuery::id`].
pub fn address<Row: MongoQuery>(
  id: &<Row::Record as Identifiable>::Id,
) -> Result<MongoPipeline, MongoQueryError> {
  let filter = identity::<Row>(id)?;
  // An address names one document by its `_id` and asks nothing else, so there is no filter to carry
  // a relation and no accumulator in the path: the predicate is built as the document it already is,
  // and the reads are the declaration's own with nothing joined onto them.
  Ok(MongoPipeline::new(
    Row::ADDRESS.table(),
    vec![stage(MATCH, Bson::Document(filter))],
    Row::READS.to_vec(),
  ))
}

/// The UNORDERED pipeline — the whole set, or any N of it.
///
/// It writes no `$sort`, which is the ruling the SQL family states: a sort nobody asked for is work
/// every document pays for, and MongoDB's own unordered semantics is the honest answer to a query
/// that named no order.
///
/// # Errors
///
/// A lowering refusal carried verbatim.
pub fn scan<Row: MongoQuery>(
  filter: Option<&RowComparison<Row>>,
  limit: Option<u32>,
) -> Result<MongoPipeline, MongoQueryError> {
  let predicate = predicate::<Row, RowComparison<Row>>(filter, None)?;
  let (stages, joined) = stages(predicate, None, None, limit.map(i64::from));
  Ok(MongoPipeline::new(
    Row::ADDRESS.table(),
    stages,
    reads::<Row>(joined),
  ))
}

/// The ORDERED pipeline, with or without a page, every term sorting by a field of the document
/// itself.
///
/// ONE function for both, because a page IS an ordered query with a ceiling and a resume predicate:
/// splitting them would give the `$sort` two renderings, which is the one thing [`MongoOrder`]
/// exists to prevent.
///
/// # The `$limit` is the page PLUS ONE, and the caller is what decides that
///
/// The extra document is *is there another one this way*, and it is trimmed before the page is
/// built. This function binds the number it is given; the lane's `window` is where the `+1` is
/// written, exactly as it is in the SQL family — so a lane that is not paging binds its own ceiling
/// unchanged.
///
/// # Errors
///
/// A lowering refusal carried verbatim, or a `skip` past what `$skip` can hold.
pub fn ordered<Row, V>(
  filter: Option<&RowComparison<Row>>,
  order: MongoOrder<'_>,
  keyset: Option<&MongoKeyset<'_, V>>,
  walk: Walk,
  limit: Option<u32>,
  skip: Option<u64>,
) -> Result<MongoPipeline, MongoQueryError>
where
  Row: MongoQuery,
  V: MongoCarrier,
{
  ordered_with::<Row, V, MongoFields>(filter, order, keyset, walk, limit, skip, MongoFields)
}

/// [`ordered`] with what each term SORTS BY written by `subjects` — the pipeline a nested order
/// renders into.
///
/// The subjects reach all three positions from here: the join stages that materialize each nested
/// term's value, the `$sort` through
/// [`sort_with`](MongoOrder::sort_with), and the resume predicate through the keyset the caller
/// already built from them. The KEYSET's own `S` is this one, which is what the shared type
/// parameter says: a keyset naming different fields from the `$sort` beside it is not a call anyone
/// can make. In practice `S` is a `&Subjects`, so the engine builds one value and every position
/// takes a copy of the borrow — the arrangement the SQL family's `Ordering` holds.
///
/// # Errors
///
/// [`ordered`]'s, which for a nested order include the relation family's own two: a hop whose target
/// is addressed by several columns, and one reaching through more references than that target's join
/// fields were minted for.
pub fn ordered_with<Row, V, S>(
  filter: Option<&RowComparison<Row>>,
  order: MongoOrder<'_>,
  keyset: Option<&MongoKeyset<'_, V, S>>,
  walk: Walk,
  limit: Option<u32>,
  skip: Option<u64>,
  subjects: S,
) -> Result<MongoPipeline, MongoQueryError>
where
  Row: MongoQuery,
  V: MongoCarrier,
  S: MongoSubjects,
{
  let address = Row::ADDRESS;
  let joins = joins(&address, order.terms(), &subjects)?;
  let predicate = predicate::<Row, MongoKeyset<'_, V, S>>(filter, keyset)?;
  let skip = skip.map(offset).transpose()?;
  let (matched, joined) = stages(
    predicate,
    Some(order.sort_with(&address, walk, &subjects)),
    skip,
    limit.map(i64::from),
  );
  // The two joining parties in the order they run, which is also the order their collections are
  // named in: the ordering's, because the `$match` reads what they materialize, and then the
  // filter's, which the same `$match` has already thinned the input for.
  let mut collections = joins.reads().to_vec();
  collections.extend(joined);
  let mut pipeline = joins.into_stages();
  pipeline.extend(matched);
  Ok(MongoPipeline::new(
    address.table(),
    pipeline,
    reads::<Row>(collections),
  ))
}

#[cfg(test)]
mod tests;