1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
//! The `pg-lo` large-tier engine (ADR-009): postgres Large Objects hold
//! the bytes; the companion table
//! `lo_entries (key bytea PRIMARY KEY, loid oid, size bigint,
//! committed_at)` is the **contract authority** — descriptorless
//! `lo_get(oid, off, len)` read windows over companion rows, one-
//! transaction puts (lo_create → chunked `lowrite` → companion-row
//! insert → COMMIT), and a held-transaction staging cursor. Feature
//! `pg-lo`, **default-off** (ADR-013 §4: `pg-lo` does not imply
//! `postgres` — the pg-lo-pure build compiles this module without
//! `feature = "postgres"`). The driver posture is `src/postgres/`'s
//! structural twin (deadpool pool over tokio-postgres, per-connection
//! `prepare_cached` discipline — statements die with sessions —
//! natively async, no `spawn_blocking` seam, no blocking I/O on the
//! async path); the large-tier bodies are `src/local.rs`'s streaming
//! twin (streaming read cursor, staged-put hygiene shapes).
//!
//! **Engine shape (POC #7 C1 is the contract).** Two artifacts: LOs
//! hold the bytes; the companion table is the addressing authority —
//! `list`, `size`, `has`, and the CAS existence probe are all
//! table-level. `pg_largeobject` is **never read for addressing**; the
//! sole sanctioned read of it is the orphan-recovery sweep's candidate
//! enumeration. `get`'s cursor closes over the companion row (oid +
//! length; content immutable, so the row is authoritative) — never
//! over a tx-scoped descriptor.
//!
//! **`lo_get` windows are the shipped get shape (ADR-009 §3, POC #7
//! C4).** Range reads are single stateless statements, pool-friendly
//! (handle-acquire p50 0.7-1.6 ms, p99 1.5-5.6 ms at ≤8 readers); a
//! read beyond the cursor's length clamps at EOF per the cursor
//! contract, and a window that comes up short *below* the length the
//! companion row claims (a corrupted half-state, which healthy stores
//! never hold) fails cleanly with an unexpected-EOF-shaped
//! [`Error::Io`]. **The held-descriptor tx posture exists as the
//! fallback only** — measured (C4): no advantage even for sequential
//! full reads, and it pins one pool connection per open handle — named
//! here as not-shipped, never coded.
//!
//! **The 64-bit-variants discipline (POC #7 C7's trap not inherited).**
//! Every offset this engine issues is 64-bit end-to-end: `lo_get`'s
//! offset parameter is `bigint` by definition, and the engine performs
//! **no int4 positioning at all** — there is no `lo_lseek` anywhere
//! (its int4 offset caps at 2 GiB; the large tier serves larger), and
//! `lowrite` rides the descriptor's implicit sequential position (one
//! writer per LO, appends only; no repositioning is ever needed). The
//! same discipline makes the descriptor a non-requirement for staged
//! writes: the descriptor advances with each `lowrite`, sequential by
//! construction.
//!
//! **512 KiB statements (POC #7 C2's measured knee).** The put cost is
//! statement count, not LOBLKSIZE (catalog rows are ceil(len/2048)
//! regardless of transport chunking): `lowrite` sweeps are flat at
//! 39-43 MB/s from 512 KiB chunks and collapse below (6 MB/s @8 KiB).
//! `LOWRITE_CHUNK` is the named shipped constant (C2's citation),
//! covering the whole-value put directly and the staging cursor's
//! append aggregation (the store core's put loop hands 64 KiB windows;
//! the engine aggregates appends into 512 KiB `lowrite` statements).
//! `READ_CHUNK`
//! bounds the `lo_get` window: larger windows don't beat the
//! server-side per-statement ceiling (~65-70 MB/s per in-flight
//! statement, C2/C3) and the int4 length parameter caps at 2 GiB.
//!
//! **Put = one transaction (ADR-009 §2).** BEGIN → `lo_create` →
//! chunked `lowrite` → companion-row insert → COMMIT, on one exclusive
//! pooled checkout. CAS first-writer-wins: the companion-table
//! existence check runs first — a re-put of an existing key is a
//! no-op tx (COMMIT of an empty transaction), never an overwrite, and
//! never a `lo_create` (no LO is born for a losing put).
//!
//! **Staged put: the held-transaction staging cursor (the engine-level
//! `put_staged` contract).** The handout opens the checkout + the tx +
//! `lo_create`; `append` rides the chunked `lowrite` aggregation; the
//! verification pass reads the staged bytes back through the read seam
//! *inside the writing tx* (read-your-own-writes: the LO's pages are
//! visible to `lo_get` from the tx that wrote them); `commit` flushes
//! the aggregation buffer, re-checks CAS existence (a re-put of an
//! existing key ROLLBACKs the staged LO away and publishes nothing —
//! first-writer-wins, `Ok`), inserts the companion row, and COMMITs;
//! `discard` ROLLBACKs. The row and the LO's pages land in one step,
//! so the crash story is structurally clean (C6: zero orphans) — and
//! every failure path converges on ROLLBACK: a failed commit
//! self-discards (there is no post-failure discard to drive), an
//! explicit discard removes LO and row-to-be together (no LO pages, no
//! row), and a cursor dropped without either is best-effort rolled
//! back by [`Drop`] (a spawned ROLLBACK when a runtime is live —
//! deadpool's `Fast` recycle never resets transaction state, so a
//! dirty checkout must never be returned silently). The held tx pins
//! one pool connection per active stage — the held-tx posture's named
//! cost, bounded by the pool size. `#[cfg(test)] fail_commit`
//! injection stands in for the media failure (the
//! mem/sqlite/postgres pattern). **Scope boundary:** the *fleet* joint
//! form below is the store-held tx edition of this staged put; the
//! engine-level staging cursor stays the non-fleet put path's own.
//!
//! **The fleet joint form — the store-held tx (ADR-012 §4's one-tx
//! ruling, wave 6; feature `postgres`).** Under the consolidation
//! posture (kv = postgres + large = pg-lo, same pg instance), a large
//! entry's content reaches the *same store-held transaction* the pin
//! row commits in: `put_joint_staged` hands a `PgLoJointStaging`
//! cursor against the held `PgJointTx` — the postgres engine's held
//! checkout, wave 4's recorded shape — and `pin_commit` (the store's
//! terminal step) publishes entry + pin row in one step over the
//! shared instance. **Named design point — the contribution's API
//! home (decided):** the *joint staging-cursor variant*, not a
//! whole-value method taking `&mut PgJointTx` — the large tier's put
//! contract is the streaming staging cursor, the fleet put driver
//! routes to the joint form exactly as the non-fleet path routes to
//! the staged form (stage-while-hashing, the key resolving at stream
//! end), and the unknown-length flow's derivation pass re-reads the
//! staged artifact through the cursor's read seam — none of which a
//! whole-value method can serve. The cursor is the staged put above
//! with inverted ownership: it owns no checkout, opens no
//! transaction, never COMMITs, never ROLLBACKs (the store core's
//! driver owns termination on every path, amendment A1), and borrows
//! the held checkout for every statement; the staged-put mechanics
//! are the landed bodies re-homed — the 512 KiB `lowrite` knee, the
//! CAS existence check, the companion-row insert, transactional LO
//! creation (the LO lives and dies with the store-held tx, C6's
//! structural zero-orphan story), read-your-own-writes staging reads.
//! **The joint CAS form:** the key resolves at publish, so
//! first-writer-wins lands there — a re-put publishes an **empty
//! contribution** (the staged LO unlinked *inside the store-held tx*:
//! the committed tx carries nothing of the losing put, zero residue,
//! and the driver's `pin_commit` upsert-renews the pin row), while a
//! concurrent CAS miss surfaces the companion key's unique violation
//! (the landed puts' posture). The store's one-tx ruling is what the
//! consolidation admits: the same held tx also carries the kv host's
//! `joint_put` row write for entries that land on that tier, one tx
//! over one instance. The engine-level held-tx staging cursor above
//! is unchanged (the non-fleet put path keeps its own tx). Nothing of
//! `EngineFleetState` exists on this engine (ADR-010 §2's correction
//! of ADR-009's ownership statement): the contribution is
//! crate-internal mechanics the store core drives.
//!
//! **Delete = one transaction** (`lo_unlink` + companion-row delete)
//! with the rows-affected discipline (`Err(Error::Missing)` when
//! absent); GC-participating — the delete window's arbitration is
//! store-core and table-level, nothing fs-specific (ADR-009 §4). `list`
//! is `SELECT key FROM lo_entries` — complete by contract, the POC's
//! exact-count gate (list == companion table after mixed ops).
//!
//! **`size` is metadata only (ADR-008's discipline):** the companion
//! row's `size` column — never LO content. The cursor's `len()` reads
//! the same column at `get` time.
//!
//! **Virgin-store posture.** The constructor creates the engine-owned
//! schema and the companion table (idempotent DDL at the constructor;
//! no migration machinery); schema-less reads map to absent/empty via
//! the SQLSTATE 42P01 discriminant (the shared
//! `crate::pg_common::is_undefined_table`), schema-less writes surface
//! the engine error — the recorded read-only split.
//!
//! **The shared-helper design point (resolved).** The postgres engine's
//! crate-internal helpers now live in `crate::pg_common`, gated
//! `cfg(any(feature = "postgres", feature = "pg-lo"))`, one module per
//! file — the pg-lo-pure build compiles without duplication:
//! `quote_identifier`, the 42P01 discriminant, the error mapping, the
//! i64-nanos timestamp encoding, the pool construction, and [`PgOpts`].
//! **PgOpts is shared** — the widening (its public re-export gate moved
//! from `postgres` to `any(postgres, pg-lo)`) was chosen over a
//! pg-lo-local opts type: the knobs are identical, the public surface
//! (one opts type per consumer story) stays single, and ADR-013 §4's
//! pg-lo-pure build carries the same opts type. The
//! `synchronous_commit` parity knob **carries**: the shipped default is
//! `on` (the kv engine's ship config); the POC #7 runs set it
//! server-side `off` to measure LO *shape* without disk fsync
//! dominating — a measurement posture, not the shipped posture.
//!
//! **LO isolation is database-level (the C7 lesson, resolved design
//! point).** Large Objects live per-database, not per-schema: the
//! schema-per-test convention does not isolate them (a `DROP SCHEMA`
//! leaves the LO's pages in `pg_largeobject`, and shared-database tests
//! race each other into spurious "large object NNNN does not exist"
//! aborts). The tests therefore run on **per-test dedicated databases**
//! (the POC's proven `open_fresh` shape): each test `CREATE DATABASE`s
//! a unique database, builds the engine inside it, and a
//! `DatabaseJanitor` drop-guard (`WITH (FORCE)`) removes the database.
//! The shared harness database (`blobs`) receives **zero LO residue by
//! construction** — no pg-lo test ever opens a pool against it.
//!
//! **Posture deltas, stated not hidden (ADR-009 §5/§6).** LO catalog
//! space is reused but never returned — the engine's inverted-fs
//! property: deleted LOs' pages free at delete-commit and are reused by
//! later puts, but `pg_largeobject`'s rel file only shrinks by
//! `VACUUM FULL`/`pg_repack` — an **operator runbook line, never
//! automatic**; monitor with `pg_total_relation_size('pg_largeobject')`.
//! Autovacuum is inherited — `pg_largeobject` is an ordinary catalog
//! heap autovacuum processes; keep it on (a deployment disabling it
//! gets the operator-visible note). Capacity facts, named not solved:
//! single-stream cached get runs 20-50x behind page-cache local fs at
//! ≥1 MiB (server-side LO page-walk + copy ceiling, ~65-70 MB/s per
//! in-flight statement); the fleet serving picture aggregates ~700 MB/s
//! at 16 readers (server-side parallelism) — the REQ-2-relevant number;
//! durable puts land 60-65 MB/s at ≥1 MiB (within 5-10% of durable
//! local fs). Deployments dominated by single-stream page-cache-bound
//! reads stay better served by `local`-on-shared-media.
//!
//! **The crate-internal orphan-recovery sweep** (`PgLoBackend::reap_orphans`,
//! ADR-009 §5, declaration-gated per ADR-016): for the legacy/bypass
//! class only — committed LOs without companion rows. Crash orphans are
//! structurally zero (C6: LO lifecycle is transactional in modern
//! postgres; kill mid-tx leaves zero pages). The sweep is gated by
//! `PgOpts.orphan_reap_permitted` (**default false**): the flag is the
//! operator's assertion that this database hosts no other large-object
//! consumer — `pg_largeobject` is database-level, not schema-level,
//! while the claim probe is companion-table-scoped, so an undeclared
//! sweep would `lo_unlink` a co-tenant consumer's committed LOs (the
//! review-001 F-4 data-loss shape); the sweep refuses with a named
//! error while the declaration is absent. ADR-016's riding fixes, all
//! landed: the `lo_entries(loid)` supporting index (the claim probe is
//! `WHERE loid = $1` per oid and the PK is `key` — bootstrap-created,
//! so existing databases gain it on next construction), and the
//! missing-LO pass (a loid vanishing between the candidate snapshot and
//! its unlink — a concurrent same-engine delete, or a foreign
//! consumer's lifecycle — rides a savepoint: the SQLSTATE 42704 pass
//! continues the sweep and is not counted reaped, instead of a plain
//! unlink's statement error aborting the whole tx). Candidates enumerate
//! `SELECT DISTINCT loid FROM pg_largeobject` as a **candidate list
//! only**, re-validated at delete time against `lo_entries.loid` (the
//! truth) — cost scales with oid count, not content size (C6: ~7 ms/oid).
//! Scope-noted under the module-level allow; the landed ops module
//! (wave 7) consumes none of it — the sweep's consumer remains the
//! embedder's operator runbook, not a store-core verb; contract-tested
//! with a planted orphan, a byte-exact survival assertion for committed
//! content, the tested refusal without the declaration, and the stale-
//! candidate pass.
//!
//! **Errors.** The engine maps its failures onto the store error
//! family: media/session/pool failure is [`Error::Io`], and
//! [`Error::Missing`] is the rows-affected delete mapping and the
//! virgin-store read-only split — no panics, no unwrap outside tests.
//! Nothing of `crate::gc::EngineFleetState` exists on this engine
//! (ADR-012 §5's pg-lo row: pin clauses **no** — fleet pins live on
//! the kv host; sweep lock **no**). The window-family seam clauses
//! (`crate::gc::EngineWindowState`) are implemented here and wired
//! by construction (landed with the constructor wiring): the store
//! core's construction call follows the one-window-host rule — active
//! on the kv=mem + large=pg-lo shape (this module's `ensure_gc_schema`
//! runs at construction; its `gc_window` hosts the durable window),
//! dormant on a dual-SQL instance (kv=sqlite/postgres + large=pg-lo:
//! the kv host's call runs, this schema carries no `gc_window`);
//! `ensure_gc_schema` creates `gc_window` only — never `gc_pins`,
//! never the embedder's liveness table.
//!
//! Tests need a live server: connection settings ride the environment
//! (`ALKBLOBS_PG_HOST/PORT/USER/PASSWORD/DB`); tests skip cleanly
//! server-less; acceptance requires green against the harness server
//! (dockerized `pglo-poc`, `postgres:16-alpine`, :15432, db `blobs`,
//! user/password `postgres`/`poc` — verified live 2026-10-12).
//!
//! The pre-split file's blanket `#![allow(dead_code)]` is gone (the
//! wave-8 split discipline: no blanket allowances survive). The
//! remaining allowance is scoped to the crate-internal orphan-recovery
//! sweep (`PgLoBackend::reap_orphans`/`reap_orphans_tx` — the
//! maintenance verb whose consumer is the embedder's operator runbook,
//! not the in-crate store paths) with the consumer named at it; the
//! joint form's feature-gated shapes need none (the `postgres` gate
//! compiles them out of the pg-lo-pure build and the fleet consumers
//! use them where they compile in); the test-only mid-driver
//! failure-injection knob needs one (its store-core consolidation
//! driver is the `all(postgres, pg-lo)` build's consumer).
//!
//! **The split layout (wave-8 structure, task `split-postgres-pg-lo`):**
//! the engine is a directory module — `staging.rs` the held-tx staging
//! cursor's bodies (with its `Drop` body) and the fleet joint form
//! (`PgLoJointStaging` + the `put_joint_staged` handout, feature
//! `postgres`), `windows.rs` the descriptorless `lo_get` window
//! machinery, the window-family seam clauses, the streaming cursor, and
//! the whole `Backend` companion-row body, `reap.rs` the orphan-recovery
//! sweep, `testing.rs` the shared test fixtures. The two
//! cross-file-shared structs (`PgLoBackend`, `PgLoStaging`) stay on the
//! module root so their private fields remain sibling-visible; the
//! public path `crate::pg_lo::PgLoCursor` is re-exported from
//! `windows.rs`.
/// The `lowrite` statement chunk — POC #7 C2's measured knee (512 KiB
/// statements; statement count, not LOBLKSIZE, is what costs: 39-43
/// MB/s flat from here, 6 MB/s @8 KiB). The store core's put loop hands
/// 64 KiB windows; both put bodies aggregate into statements of this
/// size.
pub const LOWRITE_CHUNK: usize = 512 * 1024;
/// The `lo_get` window bound — one read statement's maximum length
/// (POC #7 C2/C3: larger windows don't beat the server-side per-statement
/// ceiling; also keeps the int4 `len` parameter in domain).
pub const READ_CHUNK: usize = 512 * 1024;
/// `lo_open`'s write mode (the libpq `INV_WRITE` bit).
const INV_WRITE: i32 = 0x0002_0000;
/// The `pg-lo` engine — a deadpool pool over tokio-postgres plus the
/// quoted engine-owned companion-table reference. Cloning shares the
/// pool; the pool is the concurrency posture.
pub
/// The held-transaction staging cursor: the handout opens the checkout +
/// the tx + `lo_create` (the exclusive checkout is what carries the tx —
/// no other statement interleaves on the session); `append` rides the
/// chunked `lowrite` aggregation (the flush boundary is
/// `LOWRITE_CHUNK`; bytes below it sit in the aggregation buffer and
/// serve the read seam from memory until flushed); the verification pass
/// reads staged bytes back through the read seam inside the writing tx
/// (read-your-own-writes over `lo_get` + the unflushed tail);
/// `commit` publishes (row insert + COMMIT — row and pages land in one
/// step, C6's zero-orphan shape); every other path — explicit
/// `discard`, a failed commit, the `Drop` cancellation story —
/// converges on ROLLBACK with zero residue (no LO pages, no row).
/// The engine-owned bootstrap DDL — idempotent, no migration machinery.
/// The C1 companion-table shape; column names quote-follow the alkstore
/// hazard-list discipline (`"key"` quoted; `loid`/`size`/`committed_at`
/// non-reserved). `committed_at` is the crate-wide i64-nanos encoding
/// (`bigint`), engine-stamped at insert — not a server-default column.
/// The `(loid)` supporting index (ADR-016's riding fix): the orphan-reap
/// claim probe is `WHERE loid = $1` per oid and the PK is `key` —
/// without the index every probe seq-scans; idempotent DDL, so existing
/// databases gain it on next construction.
// consumer: `PgLoBackend::new` (the only caller); the pg-lo-only build
// compiles the body dead alongside its constructor.
/// The GC window schema's DDL — idempotent, the window-only row (ADR-012
/// §5's pg-lo row): `gc_window` keyed by the candidate key, the
/// timestamp `BIGINT` nanos. `gc_pins` is never named here (fleet pins
/// live on the kv host) and neither is the embedder's liveness table
/// (the schema-ownership rule — the sweep reads it and never owns it).
pub
pub
pub use PgLoCursor;
use ;
use crate;
use crateError;