ingraph-codegen 0.1.0

The emission engine behind ingraph-derive: a plain declaration model in, the ingraph framework faces out.
Documentation
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
//! WHAT THE ELECTION FLIP SPENDS — the two kinds' markers read off the model and into the emission.
//!
//! Beside [`tests`](super::tests) rather than inside it, on `ingraph-derive`'s `index/asked.rs`
//! precedent: one file per thing being proved, so a reader looking for what a marker BUYS finds it
//! whole rather than scattered through the emission's general suite.
//!
//! # What each kind's election reaches, and why the two differ
//!
//! A `knn` election reaches a COLUMN — the emitted marker type, the record face's `(column,
//! coordinates)` pair, and a retrieve pair named after the field. Those are proved next door, where
//! the emission's shapes live.
//!
//! A `searchable` election reaches the SUBJECT and, since the grain ruling, a DOOR of its own:
//! `FulltextRead::search` takes a `FulltextSubset` beside the query, so a per-field pair has
//! somewhere to put its answer and `FulltextColumn` is the column type it mints. What the emission
//! says is therefore both — WHICH fields the whole-subject pair answers over, in that pair's own doc
//! line, and one narrowed pair per elected field beside it.
//!
//! Most of what follows is therefore about the FULL-TEXT half, with the vector half's door count at
//! the end as its mirror. Every assertion is stated as an EXCLUSION as much as an inclusion: a
//! payload field must appear in no subject and behind no door, because *every projection* is exactly
//! the reading the marker dissolved and exactly the reading a careless emission would fall back to.

use super::{Dedup, Elected, ElectionLabel, IndexGraphql, IndexModel, Projected, Upstream};
use crate::model::{Column, GraphqlKind, NodeKind, Projection};
use crate::prefix::EmissionPrefix;

fn ident(name: &str) -> syn::Ident {
  syn::Ident::new(name, proc_macro2::Span::call_site())
}

fn column(name: &str, ty: &str) -> Column {
  Column {
    name: ident(name),
    source: ident(name),
    ty: syn::parse_str(ty).expect("a type"),
    project: Projection::Borrowed,
  }
}

fn projected(field: &str, ty: &str) -> Projected {
  Projected {
    field: ident(field),
    from: ident(field),
    ty: syn::parse_str(ty).expect("a type"),
  }
}

/// A full-text declaration with a PAYLOAD field in the middle of two searched ones.
///
/// The order is load-bearing, as it is on the vector side: an emission that named the FIRST
/// projection, or every projection, or the projections up to the first gap, would each be caught by
/// a different one of these three fields. `body` and `notes` are elected; `speaker` is not.
fn document(searchable: &[&str]) -> IndexModel {
  IndexModel {
    prefix: EmissionPrefix::default(),
    numbers: crate::index::Numbers::default(),
    field_numbers: Vec::new(),
    stores: Vec::new(),
    extension: None,
    data: false,
    keys: false,
    subscribe: false,
    graphql: Some(IndexGraphql::of(GraphqlKind::AsyncGraphql)),
    rename: None,
    slot: None,
    index: ident("SearchDocument"),
    kind: NodeKind::Fulltext,
    vis: syn::Visibility::Public(syn::token::Pub::default()),
    krate: syn::parse_quote!(::ingraph),
    source: Upstream {
      node: syn::parse_quote!(asr::Segment),
      projections: vec![
        projected("body", "String"),
        projected("speaker", "String"),
        projected("notes", "String"),
      ],
    },
    dedup: Dedup::Hash,
    embeddings: Vec::new(),
    labelled: Vec::new(),
    searchable: searchable.iter().copied().map(ident).collect(),
    // No engine named: the portable case, which declares the slots and leaves the deployment to
    // say where they are opened.
    engines: Vec::new(),
    fields: vec![
      column("body", "String"),
      column("speaker", "String"),
      column("notes", "String"),
    ],
  }
}

/// The emission's READ DOORS, walked out of the anonymous const the expansion puts its impls in —
/// the one reader both questions below are asked through.
///
/// # Two prefixes, because a read door is not always a `retrieve`
///
/// `retrieve*` is the ranked family — the two tiers, the asked road and the similar-by-example
/// sugar. `get_<field>` is the by-key read the sugar is written in terms of, and it is a door of
/// this face by every test that matters: emitted per election, named after the elected field,
/// instrumented beside the rest. A reader keyed on `retrieve` alone would call the list below
/// EXHAUSTIVE while silently skipping it, which is the failure that made this doc worth writing.
///
/// # It opens the GATE, and reads exactly ONE half
///
/// The retrieve face is emitted twice, once inside each half of the `if_tracing` / `if_no_tracing`
/// pair, so the impl is not an item of the const block any more: a gate is a macro invocation, and
/// its contents are TOKENS until the macro runs. This parses those tokens back into items and stops
/// at the first half.
///
/// One half rather than both, because of what is being asked. The halves differ in a span around
/// each door's body and in nothing else — same names, same subjects, same docs — so reading both
/// would double every answer and assert the same fact twice. That the two halves ARE the same face
/// is `tracing_tests`' subject, not this file's.
///
/// The plain-impl arm is kept beside the gated one so this reader survives the gate being removed,
/// and so it says what it is looking for rather than where the emitter happens to put it today.
fn retrieving(file: &syn::File) -> Vec<syn::ImplItemFn> {
  fn methods(items: &[syn::Item], found: &mut Vec<syn::ImplItemFn>) {
    for item in items {
      match item {
        syn::Item::Impl(declared) if declared.trait_.is_none() => {
          found.extend(declared.items.iter().filter_map(|inner| match inner {
            syn::ImplItem::Fn(method) if reads(&method.sig.ident) => Some(method.clone()),
            _ => None,
          }));
        }
        // A GATE: its body is a token stream, parsed back into the items the compiler will see.
        syn::Item::Macro(gated) => {
          if let Ok(inside) = syn::parse2::<syn::File>(gated.mac.tokens.clone()) {
            methods(&inside.items, found);
          }
        }
        _ => {}
      }
      if !found.is_empty() {
        return;
      }
    }
  }

  let mut found = Vec::new();
  for item in &file.items {
    let syn::Item::Const(anonymous) = item else {
      continue;
    };
    let syn::Expr::Block(block) = &*anonymous.expr else {
      continue;
    };
    // A gate in STATEMENT position is `Stmt::Macro`, not `Stmt::Item(Item::Macro)` — the same
    // invocation wears a different node depending on where it sits, and reading only the item form
    // is what made this walk come back empty the day the gate arrived.
    for statement in &block.block.stmts {
      match statement {
        syn::Stmt::Item(item) => methods(std::slice::from_ref(item), &mut found),
        syn::Stmt::Macro(gated) => {
          if let Ok(inside) = syn::parse2::<syn::File>(gated.mac.tokens.clone()) {
            methods(&inside.items, &mut found);
          }
        }
        _ => {}
      }
      if !found.is_empty() {
        break;
      }
    }
  }
  found
}

/// Whether an emitted method is one of the retrieve face's doors — the two prefixes the reader
/// above is keyed on, written once so the walk and its documentation cannot drift.
fn reads(name: &syn::Ident) -> bool {
  let name = name.to_string();
  name.starts_with("retrieve") || name.starts_with("get_")
}

/// The emitted file, parsed.
fn emission(model: &IndexModel) -> syn::File {
  syn::parse2(model.emit()).expect("the emission is a parsable file")
}

/// The doc lines of the retrieve methods, joined — the surface the full-text election reaches.
fn subject(model: &IndexModel) -> String {
  let file = emission(model);
  let mut said = String::new();
  for method in &retrieving(&file) {
    for attribute in &method.attrs {
      if attribute.path().is_ident("doc") {
        said.push_str(&quote::ToTokens::to_token_stream(attribute).to_string());
        said.push('\n');
      }
    }
  }
  said
}

/// The full-text pair says WHICH fields it searches, and says only the elected ones.
///
/// The exclusion is the half that reds the dissolved posture: `speaker` is a projection of this
/// declaration and appears in no subject, so an emission that went back to reading `fields` — or
/// `source.projections` — would fail here rather than pass by coincidence.
#[test]
fn the_full_text_subject_is_the_elected_fields_and_not_every_projection() {
  let said = subject(&document(&["body", "notes"]));

  assert!(
    said.contains("in `body`, `notes`"),
    "the subject names its elected fields, in declared order: {said}",
  );
  assert!(
    !said.contains("speaker"),
    "and a payload field is in no subject at all: {said}",
  );
}

/// ONE elected field is still named, which is the case a *say nothing when there is nothing to
/// choose* shortcut would have dropped.
///
/// A declaration electing one of three fields has made a choice, and it is the same choice a
/// declaration electing two made; the reader of the emitted doc needs it in both.
#[test]
fn a_single_election_is_named_too() {
  let said = subject(&document(&["notes"]));

  assert!(said.contains("in `notes`"), "{said}");
  assert!(!said.contains("`body`"), "{said}");
  assert!(!said.contains("speaker"), "{said}");
}

/// The election NOW MOVES THE NAMES, and this assertion is the one its own old form named as the
/// thing to revisit.
///
/// # It is the same law respelled, and the sentence it used to carry is why
///
/// > A per-field door would have to tell the engine which field answered, and
/// > `FulltextRead::search` takes a slot, a scope and a string. Suffixing the names here would
/// > emit two doors that could only answer the same thing — so what the marker changes is the
/// > subject, and the day the engine grammar gives a slot a column is the day this assertion is the
/// > one to revisit.
///
/// That day is the grain ruling. `search` takes a `FulltextSubset` now, so a suffixed door has
/// somewhere to put its answer and two of them no longer answer the same thing. The law the old
/// assertion actually held — *a door's name must be backed by a narrowing the seam can carry* — is
/// unchanged, and it is what the new shape is measured against.
///
/// # N + 1, and the unsuffixed pair is not an alias
///
/// The whole-subject pair stays, and comes FIRST. It asks `All`, which is *whatever the slot holds*
/// and is a different question from *this field* — a slot may hold columns the declaration never
/// elected. So a declaration electing ONE field still emits both, which is where an emission
/// collapsing the single-election case into a bare pair would fail: the vector kind's
/// no-name-moves-with-cardinality rule holds on this side because the whole-subject pair was never
/// one of the per-field ones.
#[test]
fn the_full_text_election_grows_a_pair_beside_the_whole_subject_one() {
  assert_eq!(
    doors(&document(&["body", "notes"])),
    [
      "retrieve_keys",
      "retrieve",
      "retrieve_keys_by_body",
      "retrieve_by_body",
      "retrieve_keys_by_notes",
      "retrieve_by_notes",
    ],
    "two elections: the whole-subject pair, then one pair each in declared order",
  );
  assert_eq!(
    doors(&document(&["notes"])),
    [
      "retrieve_keys",
      "retrieve",
      "retrieve_keys_by_notes",
      "retrieve_by_notes",
    ],
    "and ONE election is still two pairs — the whole-subject door is not the field's door",
  );
  assert!(
    !doors(&document(&["body", "notes"]))
      .iter()
      .any(|door| door.contains("speaker")),
    "a payload field is behind no door, which is the exclusion the subject states in prose",
  );
}

/// A per-field door NARROWS, and the narrowing is a one-column `These` the method MINTS.
///
/// The mutation this pins is the one an emission takes by accident: reusing the whole-subject arm
/// for every subject, so `retrieve_by_notes` compiles, is named after a field, and asks `All` — a
/// door whose NAME says one column and whose CALL says every column, which is the standing lie the
/// door family exists to refuse. The column name is asserted per door, because an emission reading
/// the first searchable field for every pair would put `body` in both.
#[test]
fn a_per_field_door_asks_its_own_column_and_the_whole_subject_door_asks_all() {
  let file = emission(&document(&["body", "notes"]));
  let body = |name: &str| {
    retrieving(&file)
      .into_iter()
      .find(|method| method.sig.ident == name)
      .map(|method| quote::ToTokens::to_token_stream(&method.block).to_string())
      .unwrap_or_else(|| panic!("`{name}` is emitted"))
  };

  let whole = body("retrieve_keys");
  assert!(
    whole.contains("FulltextSubset :: All"),
    "the whole-subject door asks over everything the slot holds: {whole}",
  );
  assert!(!whole.contains("These"), "and narrows nothing: {whole}");

  for (door, column, other) in [
    ("retrieve_keys_by_body", "body", "notes"),
    ("retrieve_keys_by_notes", "notes", "body"),
  ] {
    let said = body(door);
    assert!(
      said.contains("FulltextSubset :: These"),
      "`{door}` narrows: {said}",
    );
    assert!(
      said.contains(&format!("FulltextColumn :: new (\"{column}\")")),
      "`{door}` names its own column: {said}",
    );
    assert!(
      !said.contains(&format!("\"{other}\"")),
      "`{door}` names no other election's: {said}",
    );
  }
}

/// A VECTOR declaration with a PAYLOAD embedding between two elected ones.
fn embeddings(elected: &[&str]) -> IndexModel {
  IndexModel {
    prefix: EmissionPrefix::default(),
    stores: Vec::new(),
    extension: None,
    data: false,
    keys: false,
    subscribe: false,
    index: ident("SegmentEmbedding"),
    kind: NodeKind::Vector,
    source: Upstream {
      node: syn::parse_quote!(asr::Segment),
      projections: vec![
        projected("image_embedding", "Embedding<f32, 4>"),
        projected("caption_embedding", "Embedding<f32, 4>"),
        projected("audio_embedding", "Embedding<f32, 4>"),
      ],
    },
    embeddings: elected
      .iter()
      .map(|field| {
        Elected::new(
          ident(field),
          ElectionLabel::Embedding,
          syn::parse_str("Embedding<f32, 4>").expect("a type"),
          None,
        )
      })
      .collect(),
    labelled: Vec::new(),
    searchable: Vec::new(),
    fields: vec![
      column("image_embedding", "Embedding<f32, 4>"),
      column("caption_embedding", "Embedding<f32, 4>"),
      column("audio_embedding", "Embedding<f32, 4>"),
    ],
    ..document(&[])
  }
}

/// Every emitted read door's name, in emission order.
fn doors(model: &IndexModel) -> Vec<String> {
  let file = emission(model);
  retrieving(&file)
    .iter()
    .map(|method| method.sig.ident.to_string())
    .collect()
}

/// A PAYLOAD embedding gets no door of its own, which is the other half of the exclusion the
/// full-text subject states in prose.
///
/// The payload is declared FIRST here, so an emission walking `fields` instead of `embeddings`
/// would put its doors at the head of this list rather than leaving it out.
///
/// The two BY-KEY doors ride the same exclusion, and that is the sharper half of the claim: the
/// by-key read is a capability of the SEAM rather than of a declared word, so nothing stops an
/// emission from offering *this entity's stored `image_embedding`* — except that a payload column is a
/// column no engine holds a corpus for, so neither the read nor the search over it means anything.
#[test]
fn a_payload_embedding_is_behind_no_door() {
  assert_eq!(
    doors(&embeddings(&["caption_embedding", "audio_embedding"])),
    [
      "retrieve_keys_by_caption_embedding",
      "retrieve_by_caption_embedding",
      "get_caption_embedding",
      "retrieve_similar_by_caption_embedding",
      "retrieve_keys_by_audio_embedding",
      "retrieve_by_audio_embedding",
      "get_audio_embedding",
      "retrieve_similar_by_audio_embedding",
    ],
    "one QUAD per election, in declared order, and nothing at all for `image_embedding`",
  );
}