1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
//! The SINK a full-text converter writes into — one `push` per SEGMENT of the document.
//!
//! A field whose value is not already text reaches a full-text index through a converter the index
//! node's declaration names, and that converter takes a [`BoxWriter`] rather than returning a string:
//!
//! ```
//! use ingraph::BoxWriter;
//!
//! struct Label(&'static str);
//!
//! fn join_labels(out: &mut BoxWriter, labels: &Vec<Label>) {
//! for label in labels {
//! out.push(label.0);
//! }
//! }
//!
//! let mut out = BoxWriter::new();
//! join_labels(&mut out, &vec![Label("red car"), Label("blue sky")]);
//! assert_eq!(out.segments().collect::<Vec<_>>(), ["red car", "blue sky"]);
//! ```
//!
//! It names no engine, and nothing gates it. A declaration names its converter, so the signature an
//! author writes has to be spellable in a build with no backend at all — the type is reached from
//! the DECLARING side, which has no features of the framework's to speak of.
//!
//! # Why a sink, and not `-> String`
//!
//! **Because the SEGMENTATION has to survive, and a returned string destroys it.** A positional
//! full-text engine separates the VALUES of one field so that an exact phrase cannot match across the
//! join. Tantivy 0.26.1 is the measured case: `src/postings/postings_writer.rs` declares
//! `const POSITION_GAP: u32 = 1` (line 18) and, after each value's token stream, advances
//! `indexing_position.end_position = end_position + POSITION_GAP` (line 158) — so the next value's
//! first token sits one position PAST where a continuation would have been, and a phrase query
//! spanning the boundary matches nothing. A converter returning one `String` collapses every value
//! into a single append, and then a phrase CAN match the tail of one label and the head of the next: a
//! hit for a phrase the corpus does not contain. Pushing per segment is what keeps that impossible,
//! and it puts the choice of where the boundaries are with the author, who is the only one who knows
//! whether two values are one utterance or two.
//!
//! **And because the signature has to be NAMABLE.** `fn(&mut BoxWriter, &T)` is concrete, so
//! generated code can write the call and the compiler can check the declared path against it at the
//! site. Returning `impl Iterator<Item = &str>` — the shape that would preserve segmentation without a
//! sink — has no spelling an expansion can write; returning `String` has one, and pays an allocation
//! per call for the collapse described above.
//!
//! # What it is not
//!
//! Nothing hands the buffer out as ONE string. That accessor would be exactly the concatenation the
//! segmentation exists to prevent, and it would read like "the document" while being a different
//! document from the one the converter described. [`BoxWriter::segments`] is the whole read face.
//!
//! The name is the ONE-BOX destination: today a document has a single text field, so every segment a
//! converter pushes lands in the same box and a writer needs no field to address. Per-field
//! (`named`) documents are a declaration this grammar does not have yet, and until it does there is
//! nothing for a second destination to be.
//!
//! # The two builtins
//!
//! Most converters are one of two loops, so both are here as methods rather than written out by
//! every author: [`BoxWriter::push_each`] and [`BoxWriter::push_joined`]. The pair is the WHOLE
//! choice a segmented sink offers, and it is the choice the header is about — `each` keeps the values
//! apart so a phrase cannot span two of them, `join` puts them in one value so it can, through a
//! separator the caller names. A hand-written converter is for everything else, and neither builtin
//! can do anything one could not.
//!
//! The index node's derive HAS that grammar now, and the two loops are what its two shorthand words
//! spend: `multi` is [`push_each`](BoxWriter::push_each) and `join = "…"` is
//! [`push_joined`](BoxWriter::push_joined). The pair was written before anything picked between
//! them and is unchanged by being picked between — which is the whole of what *expressible before
//! anything consumes it* was worth.
//!
//! # The THREE ROADS, and [`Searchable`] is the one that can refuse
//!
//! A `searchable` field reaches this sink one of three ways: the value is text already (IDENTITY),
//! a builtin walks a multi-valued field (SHORTHAND), or the declaration names a converter
//! (`from`/`try_from`). Each road carries its own bound, so a field the road does not fit is a
//! compile error at the field rather than a promise the parser could not check — and the identity
//! one is spelled through [`Searchable`] rather than through [`push`](BoxWriter::push) so that the
//! refusal can name the other two.
/// The sink a full-text converter pushes SEGMENTS of text into.
///
/// One [`push`](Self::push) is one segment. Segments are kept apart rather than joined, because a
/// positional engine's value boundary is what stops a phrase query matching across two of them — the
/// module header carries the measurement.
///
/// Backed by ONE `String` plus each segment's end offset, not a `Vec<String>`: a push is a byte copy
/// into a buffer that grows amortized, so a converter over N values pays no per-value allocation.
/// That is the half of the "not `-> String`" argument that is about cost; the other half is about
/// correctness and is the reason this type exists at all.
/// THE IDENTITY ROAD: a value that is ALREADY the document text.
///
/// One segment, pushed as it stands. It is what a `searchable` field takes when its engine's block
/// names no road at all, and it is the road that has to be able to say NO — a field whose value is
/// not text has no identity road, and the useful answer names the two roads that would have carried
/// it.
///
/// # It is a bound, and the bound IS the check
///
/// The derive performs no type test on a `searchable` field and must not: it has no type
/// information, and the one check a parser could make is a spelling allowlist, which false-refuses
/// `SmolStr`, a newtype, an alias and `Vec<String>` alike — and this crate calls a false refusal a
/// defect of its own. So the road is checked where the types are known. A field the identity road
/// does not fit is `E0277` at that field's own type token, carrying the note below.
///
/// # Blanket over `AsRef<str>`, and that is where the surface stays open
///
/// Nothing implements this trait by hand and nothing needs to: the blanket covers every type that
/// can already be read as text, which is the same seam [`BoxWriter::push`] takes. A domain newtype
/// joins the road by implementing `AsRef<str>` — one impl on the writer's own type, no orphan to
/// argue with — and the blanket carries it here. Publishing a second membership trait beside the
/// std one would be a second list to keep in step with the first.