evidencebook 0.0.23

Evidence book: records, epistemic types, typed links, retention, disclosure, index classes, request answers and reconcile/close over an embedded Checkpointed Local Log -- a Rust implementation of the Evidence Layer semantics.
Documentation
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
//! The shelf: one subject key's books in one SQLite file (evidencebook#254
//! §2), built with the cargo feature `shelf`.
//!
//! The file holds CLL's tables (one log per book id, the cadence log and
//! every host log), capsule-emit's artifact store (the originals of every
//! book and the records of every host log, under one namespace) and the
//! shelf's own tables (`book_*`, `cadence_*`, `shelf_*`),
//! exactly as the Go shelf writes them, so Go, Python and Rust writers
//! continue each other's files. `testdata/shelf` holds the byte vectors, and
//! `test/interop/shelf` (Go), `test/interop/python-evidencebook/shelf.py`
//! (Python) and `rust/interop`'s `shelf` bin (Rust) run the storage
//! interlock, that check it.
//!
//! - [`Shelf`] opens the file under the shelf's writer lock or a lock the
//!   host already holds ([`HeldLock`]), and hands out one [`ShelfBook`] per
//!   log id ([`Shelf::open`]).
//! - A [`HostLog`] is a log on the shelf whose records the host seals
//!   itself, on the shelf's CLL backend and artifact store, never converted
//!   into book records ([`Shelf::host_log`], [`Shelf::legacy`]).
//! - [`Cadence`] runs the heartbeat ticks that commit every live log's
//!   checkpoint as a salted leaf of a sparse tree whose root alone reaches
//!   the witness; after [`Cadence::migrate`] each tick is a stamp that
//!   appends book facts to the cadence log instead (evidencebook#461).
//! - [`Cadence::handover`] hands the whole shelf log over to a reader the
//!   holder chooses, and [`verify_handover`] replays it (evidencebook#461
//!   §2.7).
//!
//! **Readers without the lock.** Another process may read the file while a
//! writer holds the shelf: the file is in WAL mode, so a reader that opens
//! a log with checkpointed-local-log's `log::sqlite::SqliteStore::open` (which
//! validates and writes nothing) or the artifact store on the same file sees
//! every committed write and never waits on, or takes, the writer lock. Such a
//! reader must not write. The shelf leaves every table it does not own (CLL's,
//! the artifact store's, and any host table) as it finds it.
//!
//! **Platforms.** The shelf's own lock is a unix `flock`; off unix
//! [`Shelf::open_sqlite`] opens only on a host-held lock ([`HeldLock`]),
//! which takes no lock and never opens a lock file.
//!
//! This module is the one place the crate's public API names a `cll` type:
//! a host log hands the host the log's CLL backend ([`HostLogStores`]),
//! since the host's own log layer runs on it (§2.7).

mod book;
mod cadence;
mod chain;
mod gojson;
mod handover;
mod handover_tie;
#[cfg(test)]
mod handover_tie_tests;
mod handover_times;
mod held;
mod history;
#[cfg(test)]
mod hooks_tests;
mod import;
mod lock;
mod log;
mod push;
mod random;
mod receive;
#[cfg(test)]
mod receive_tests;
mod record_time;
#[cfg(test)]
mod record_time_tests;
mod records;
mod sent;
mod stamp;
mod stamp_proof;
mod store;
mod time;
mod tree;

pub use book::{BookEntry, BookRecord, Indexes, ShelfBook, RECORD_HEADER_MEMBER};
pub use cadence::{
    Cadence, CadenceConfig, CadenceTick, TickResult, WitnessCheck, WitnessReceipt, WITNESS_INVALID,
    WITNESS_NOT_CHECKED, WITNESS_VERIFIED,
};
pub use chain::CADENCE_EXTENSION;
pub use handover::{
    verify_handover, ConsistencyEvidence, Handover, HandoverCheckpoint, HandoverConsistency,
    HandoverDocument, HandoverLatest, HandoverResult,
};
pub use handover_tie::HandoverTie;
pub use handover_times::{verify_handover_times, HandoverTimes};
pub use held::HeldProof;
pub use history::StandardHistoryAnchor;

/// The log id of the shelf's journal ([`Shelf::journal`]).
pub const JOURNAL_LOG_ID: &str = "journal";
pub use import::LegacyCadence;
pub use log::{Checkpoint, SUBSTRATE_KIND_CLL};
pub use push::{
    push_a2a_extension, PushA2a, PushA2aRequest, PushA2aResponse, PushMcp, A2A_PROTOCOL_VERSION,
    A2A_PUSH_EXTENSION_URI, MCP_PROTOCOL_VERSION, MCP_TOOL_RECORD_PUSH,
};
pub use random::{RandomSource, SeededStream, SystemRandom};
pub use receive::{
    CountedPrefix, Evidence, HeldRecord, ProofOutcome, ReceiveBounds, ReceiveChanges,
    ReceiveConfig, ReceiveError, ReceiveProfile, ReceiveResult, ReceivedRecord, RecordOutcome,
    EVIDENCE_CHECKPOINT, EVIDENCE_CHECKPOINT_RECORDS, EVIDENCE_FORK,
};
pub use records::{ShelfRecord, ShelfRecords};
pub use sent::{SealedSent, SentCapsule};
pub use stamp::{AudienceFact, BatchEntry, BookStatus, StampBatch, StatusInput, StatusValues};
pub use stamp_proof::{verify_stamp_proof, InclusionEvidence, StampProof, VerifiedStampProof};
pub use store::{StoredRecord, SHELF_VERSION, SHELF_VERSION_FACTS};
pub use time::Time;
pub use tree::{
    cadence_empty, cadence_filler_leaf, cadence_fold, cadence_leaf, cadence_tree, derived_fillers,
    padding_record_id, TickLeaf, TickLeaves, CADENCE_DEPTH,
};

use self::lock::FileLock;
use self::log::Log;
use self::random::Draws;
use self::store::{KIND_BOOK, KIND_HOST, KIND_JOURNAL};
use crate::substrate::Signer;
use capsule_emit_artifact::Store as ArtifactStore;
use ed25519_dalek::SigningKey;
use rusqlite::Connection;
use std::collections::HashMap;
use std::path::{Path, PathBuf};
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, Mutex, MutexGuard};
use std::time::{Duration, Instant};

/// Why a shelf operation failed.
#[derive(Debug, thiserror::Error)]
pub enum ShelfError {
    /// The call or its configuration is not valid.
    #[error("invalid: {0}")]
    Invalid(String),
    /// Stored state is inconsistent.
    #[error("corrupt: {0}")]
    Corrupt(String),
    /// What was asked for does not exist.
    #[error("not found: {0}")]
    NotFound(String),
    /// A position is not covered by the checkpoint asked about.
    #[error("not covered: {0}")]
    NotCovered(String),
    /// The file's version, or the platform, is not one this build supports.
    #[error("unsupported: {0}")]
    Unsupported(String),
    /// An answer would carry more items than the cost policy allows
    /// ([`crate::respond::DEFAULT_MAX_ANSWER_ITEMS`]), or a received record
    /// is past one of the store's bounds ([`ReceiveBounds`]).
    #[error("over the item limit: {0}")]
    OverLimit(String),
    /// The shelf was released.
    #[error("the shelf is released")]
    Released,
    /// A received record or stamp proof whose sender may not push it
    /// ([`Shelf::receive`]): nothing of it is stored.
    #[error("refused: {0}")]
    Refused(String),
    /// A received record deleted under the holder's retention: the store
    /// keeps its tombstone, so it is deleted, not missing.
    #[error("deleted: {0}")]
    Deleted(String),
    /// A payload whose record's retention state yields no bytes now
    /// (`WITHHELD` or `DELETED`; Go `ErrUnavailable`): a withheld payload
    /// may become available again ([`ShelfBook::resolve_payload`]).
    #[error("payload unavailable: {0}")]
    Unavailable(String),
    /// A received offer refused because the holder's books kept changing
    /// while room was made for it ([`Shelf::receive`]): sending it again
    /// later may succeed.
    #[error("busy; try again: {0}")]
    Busy(String),
    /// The receive profile ([`ReceiveProfile`]) failed.
    #[error("receive profile: {0}")]
    Profile(String),
    /// Another writer holds the shelf's lock past the caller's deadline.
    #[error("{0}: another writer holds it")]
    LockHeld(String),
    /// The CLL log failed.
    #[error("log: {0}")]
    Log(String),
    /// The artifact store failed.
    #[error("originals: {0}")]
    Artifact(String),
    /// The random source failed.
    #[error("{0}")]
    Random(String),
    /// Sealing a record failed.
    #[error(transparent)]
    Seal(#[from] crate::seal::SealRecordError),
    /// The shelf's SQLite file failed.
    #[error("sqlite: {0}")]
    Sqlite(#[from] rusqlite::Error),
    /// A file failed.
    #[error("io: {0}")]
    Io(#[from] std::io::Error),
}

/// A writer lock the host holds on the shelf's store under its own rules
/// (evidencebook#254 §2.4). [`HeldLock::held`] reports `Ok` while the host
/// holds it; the shelf asks once, when it opens, and never releases it.
pub trait HeldLock: Send + Sync {
    /// `Ok` while the host holds the lock.
    fn held(&self) -> Result<(), String>;
}

/// A shared lock is the lock it shares, so a host can keep its own handle
/// and give the shelf a clone.
impl<T: HeldLock + ?Sized> HeldLock for Arc<T> {
    fn held(&self) -> Result<(), String> {
        (**self).held()
    }
}

/// The shelf's clock.
pub type Clock = Arc<dyn Fn() -> Time + Send + Sync>;

/// Which authenticated indexes each book gets, by log id.
pub type IndexChoice = Arc<dyn Fn(&str) -> Indexes + Send + Sync>;

/// What a shelf is opened with (Go `ShelfConfig`). The signer, checkpoint
/// key, each book's indexes, the originals' namespace and the cadence's
/// configuration are host parameters: every process that writes one shelf,
/// in any language, must use the same values (§2.2).
pub struct ShelfConfig {
    /// The SQLite file.
    pub path: PathBuf,
    /// The writer lock file; `None` is `<path>.lock`. A host whose other
    /// writers already lock a file of their own passes that one, so they
    /// and the shelf exclude each other. Must be `None` with `held_lock`.
    pub lock_path: Option<PathBuf>,
    /// The store's writer lock when the host already holds it: the shelf
    /// then takes no lock of its own, and opens on any platform.
    pub held_lock: Option<Box<dyn HeldLock>>,
    /// How long to wait for another holder of the shelf's own lock;
    /// `None` waits until it is free.
    pub lock_timeout: Option<Duration>,
    /// The shelf's one random source; `None` is the system CSPRNG.
    pub random: Option<Box<dyn RandomSource>>,
    pub operator: String,
    /// The capsules' `developer`; empty is `evidencebook`.
    pub developer: String,
    /// Seals every book's records.
    pub signer: Arc<dyn Signer + Send + Sync>,
    /// Signs every log's checkpoints.
    pub checkpoint_key: SigningKey,
    /// The shelf's one artifact store, which must trust `signer`'s key: the
    /// originals of every book and the records of every host log. For the
    /// contract's file it is capsule-emit's SQLite store on `path` (§2.5).
    pub originals: Arc<dyn ArtifactStore + Send + Sync>,
    /// The clock; `None` is the system clock.
    pub now: Option<Clock>,
    /// Each book's authenticated indexes; `None` gives every book the
    /// defaults.
    pub authenticated_indexes: Option<IndexChoice>,
    /// The header `correlation` members the profile writes, in join order,
    /// each a member the record header schema defines (its `correlation` is
    /// closed, and a Go shelf refuses to reopen a book whose header carries
    /// any other). [`ShelfBook::append`] refuses a key not listed. A book's
    /// [`crate::store::RecordSource::correlation`] holds a record carrying
    /// the first member under it alone, and any other record under its
    /// `subject_ref`. Go's book holds a record under the record header
    /// schema's first correlation member, so a profile that lists the
    /// schema's members in its order answers a correlation subject as Go
    /// does on the same file. The names are the profile's, never this
    /// crate's.
    pub correlation_members: Vec<String>,
    /// The store of what the other side pushes ([`Shelf::receive`]): the
    /// profile that reads its records, the store's bounds and the holder's
    /// retention.
    pub receive: ReceiveConfig,
}

/// What every book and host log of a shelf shares.
pub(crate) struct ShelfShared {
    pub(crate) path: PathBuf,
    pub(crate) operator: String,
    pub(crate) developer: String,
    pub(crate) signer: Arc<dyn Signer + Send + Sync>,
    pub(crate) checkpoint_key: SigningKey,
    pub(crate) originals: Arc<dyn ArtifactStore + Send + Sync>,
    now: Option<Clock>,
    authenticated: Option<IndexChoice>,
    pub(crate) correlation_members: Vec<String>,
    pub(crate) receive: ReceiveConfig,
    /// Serializes [`Shelf::receive`] and [`Shelf::forget`], which call the
    /// receive profile outside the shelf's lock so the profile may read the
    /// shelf.
    pub(crate) receive_mu: Mutex<()>,
    /// Mirrors `released` for the received store's readers, which never take
    /// the shelf's lock.
    pub(crate) closed: AtomicBool,
    db: Mutex<Connection>,
}

impl ShelfShared {
    pub(crate) fn now(&self) -> Time {
        self.now.as_ref().map_or_else(Time::now, |f| f())
    }

    pub(crate) fn db(&self) -> MutexGuard<'_, Connection> {
        self.db.lock().unwrap_or_else(|p| p.into_inner())
    }
}

/// A host log's state on its shelf.
pub(crate) struct HostCore {
    pub(crate) id: String,
    pub(crate) log: Arc<Log>,
    pub(crate) in_cadence: bool,
}

pub(crate) struct ShelfState {
    pub(crate) books: HashMap<String, ShelfBook>,
    pub(crate) hosts: HashMap<String, Arc<HostCore>>,
    /// The cadence log's id and log, once a cadence is opened.
    pub(crate) cadence: Option<(String, Arc<Log>)>,
    pub(crate) random: Draws,
    pub(crate) released: bool,
    lock: Option<FileLock>,
}

pub(crate) struct ShelfInner {
    pub(crate) shared: Arc<ShelfShared>,
    pub(crate) state: Mutex<ShelfState>,
}

/// One subject key's books in one SQLite file (Go `Shelf`). The shelf holds
/// the writer lock from [`Shelf::open_sqlite`] to [`Shelf::release`] and is
/// the file's only writer. Handles are cheap clones of one shelf.
///
/// Each book keeps its own lock; operations across books, such as a
/// cadence tick, take the shelf's.
#[derive(Clone)]
pub struct Shelf(pub(crate) Arc<ShelfInner>);

/// A handle on a shelf that does not keep it alive ([`Shelf::downgrade`]):
/// what a [`ReceiveProfile`] the shelf owns holds of the shelf, so the two
/// do not keep each other alive.
#[derive(Clone)]
pub struct WeakShelf(std::sync::Weak<ShelfInner>);

impl WeakShelf {
    /// The shelf, while any [`Shelf`] handle on it lives.
    pub fn upgrade(&self) -> Option<Shelf> {
        self.0.upgrade().map(Shelf)
    }
}

impl Shelf {
    /// Take the writer lock (waiting up to `lock_timeout`) or check the
    /// host's held lock, and open the shelf's file, creating its tables when
    /// absent. A file whose `shelf_meta` version this build does not know is
    /// refused; one with no row is a fresh file or a shelf from before the
    /// table existed, and gets version 1 with `shelf_logs` filled from what
    /// it holds (§2.2).
    pub fn open_sqlite(cfg: ShelfConfig) -> Result<Shelf, ShelfError> {
        if cfg.path.as_os_str().is_empty() || cfg.operator.is_empty() {
            return Err(ShelfError::Invalid("path and operator are required".into()));
        }
        let lock = match &cfg.held_lock {
            Some(held) => {
                if cfg.lock_path.is_some() {
                    return Err(ShelfError::Invalid(
                        "a shelf on a host-held lock takes no lock path".into(),
                    ));
                }
                held.held().map_err(|e| {
                    ShelfError::Invalid(format!("the host's writer lock is not held: {e}"))
                })?;
                None
            }
            None => {
                let path = cfg.lock_path.clone().unwrap_or_else(|| {
                    let mut p = cfg.path.clone().into_os_string();
                    p.push(".lock");
                    PathBuf::from(p)
                });
                Some(FileLock::acquire(
                    &path,
                    cfg.lock_timeout.map(|t| Instant::now() + t),
                )?)
            }
        };
        let mut conn = Connection::open(&cfg.path)?;
        conn.busy_timeout(Duration::from_millis(5000))?;
        conn.pragma_update(None, "journal_mode", "WAL")?;
        conn.pragma_update(None, "synchronous", "FULL")?;
        store::init(&mut conn)?;
        let shared = Arc::new(ShelfShared {
            path: cfg.path,
            operator: cfg.operator,
            developer: cfg.developer,
            signer: cfg.signer,
            checkpoint_key: cfg.checkpoint_key,
            originals: cfg.originals,
            now: cfg.now,
            authenticated: cfg.authenticated_indexes,
            correlation_members: cfg.correlation_members,
            receive: cfg.receive,
            receive_mu: Mutex::new(()),
            closed: AtomicBool::new(false),
            db: Mutex::new(conn),
        });
        Ok(Shelf(Arc::new(ShelfInner {
            shared,
            state: Mutex::new(ShelfState {
                books: HashMap::new(),
                hosts: HashMap::new(),
                cadence: None,
                random: Draws(cfg.random.unwrap_or_else(|| Box::new(SystemRandom))),
                released: false,
                lock,
            }),
        })))
    }

    /// A handle on this shelf that does not keep it alive.
    pub fn downgrade(&self) -> WeakShelf {
        WeakShelf(Arc::downgrade(&self.0))
    }

    pub(crate) fn lock(&self) -> MutexGuard<'_, ShelfState> {
        self.0.state.lock().unwrap_or_else(|p| p.into_inner())
    }

    pub(crate) fn shared(&self) -> &Arc<ShelfShared> {
        &self.0.shared
    }

    /// The shelf's file.
    pub fn path(&self) -> &Path {
        &self.0.shared.path
    }

    /// The book whose log id is `log_id`, created on first use: the same
    /// book for the same id for the shelf's lifetime. A host log's id is
    /// refused, in this process or as `shelf_logs` records it, and so is the
    /// journal's ([`Shelf::journal`] opens it).
    pub fn open(&self, log_id: &str) -> Result<ShelfBook, ShelfError> {
        let mut state = self.lock();
        if state.released {
            return Err(ShelfError::Released);
        }
        if store::log_kind(&self.shared().db(), log_id)?.as_deref() == Some(KIND_JOURNAL) {
            return Err(ShelfError::Invalid(format!(
                "{log_id} is the shelf's journal; open it with journal"
            )));
        }
        self.open_locked(&mut state, log_id)
    }

    /// The shelf's journal (evidencebook#461 §3; Go `Journal`): a book of
    /// kind `journal` whose log id is [`JOURNAL_LOG_ID`], one per shelf,
    /// opened on first use, whose records are full records. Its facts are
    /// journal facts: a whole-shelf reader sees that it changed at a stamp,
    /// never its size, and its openings go only to the owner and an auditor
    /// it appoints. It needs a shelf of book facts ([`Cadence::migrate`]),
    /// and it refuses a shelf on which a book or host log already has its
    /// log id. A file whose `shelf_logs` table predates the kind gets the
    /// table rebuilt, with the same rows, when the journal is first opened.
    pub fn journal(&self) -> Result<ShelfBook, ShelfError> {
        let mut state = self.lock();
        if state.released {
            return Err(ShelfError::Released);
        }
        let shared = self.shared().clone();
        if let Some(book) = state.books.get(JOURNAL_LOG_ID) {
            // Opened before: as the journal, or as a book that holds its id.
            if store::log_kind(&shared.db(), JOURNAL_LOG_ID)?.as_deref() == Some(KIND_JOURNAL) {
                return Ok(book.clone());
            }
            return Err(ShelfError::Invalid(format!(
                "{JOURNAL_LOG_ID} is already a book, not the journal"
            )));
        }
        if stamp::migration_state(&shared.db())? != stamp::MIGRATION_DONE {
            return Err(ShelfError::Invalid(
                "the journal needs a shelf of book facts; call migrate".into(),
            ));
        }
        if state.hosts.contains_key(JOURNAL_LOG_ID) {
            return Err(ShelfError::Invalid(format!(
                "{JOURNAL_LOG_ID} is a host log, not the journal"
            )));
        }
        if state
            .cadence
            .as_ref()
            .is_some_and(|(id, _)| id == JOURNAL_LOG_ID)
        {
            return Err(ShelfError::Invalid(format!(
                "{JOURNAL_LOG_ID} is the cadence log, not the journal"
            )));
        }
        {
            let mut db = shared.db();
            let tx = store::write_tx(&mut db)?;
            let records = store::record_count(&tx, JOURNAL_LOG_ID)?;
            let kind = store::log_kind(&tx, JOURNAL_LOG_ID)?;
            if kind.as_deref() != Some(KIND_JOURNAL) && (kind.is_some() || records > 0) {
                return Err(ShelfError::Invalid(format!(
                    "{JOURNAL_LOG_ID} is already a {} log, not the journal",
                    kind.as_deref().unwrap_or(KIND_BOOK)
                )));
            }
            // A log with entries and no kind (the cadence log, which
            // shelf_logs never records, or one this build does not know) is
            // not the journal either: claiming it would leave its owner
            // refused on the next open.
            if kind.is_none() {
                let entries: i64 = tx.query_row(
                    "SELECT COUNT(*) FROM cll_entries WHERE log_id = ?",
                    [JOURNAL_LOG_ID],
                    |r| r.get(0),
                )?;
                if entries > 0 {
                    return Err(ShelfError::Invalid(format!(
                        "{JOURNAL_LOG_ID} already holds another log's entries, not the journal"
                    )));
                }
            }
            store::allow_journal_kind(&tx)?;
            store::claim_kind(&tx, JOURNAL_LOG_ID, KIND_JOURNAL)?;
            tx.commit()?;
        }
        self.open_locked(&mut state, JOURNAL_LOG_ID)
    }

    /// The shelf's journal when the shelf has one, `None` otherwise (Go
    /// `ExistingJournal`). Unlike [`Shelf::journal`] it never creates it, so
    /// a reader (a report) changes nothing.
    pub fn existing_journal(&self) -> Result<Option<ShelfBook>, ShelfError> {
        let mut state = self.lock();
        if state.released {
            return Err(ShelfError::Released);
        }
        self.existing_journal_locked(&mut state)
    }

    /// The shelf's journal if the shelf has one, `None` otherwise: a stamp
    /// hands it to a [`BookStatus`] and must not create it.
    pub(crate) fn existing_journal_locked(
        &self,
        state: &mut ShelfState,
    ) -> Result<Option<ShelfBook>, ShelfError> {
        let shared = self.shared().clone();
        if stamp::migration_state(&shared.db())? != stamp::MIGRATION_DONE {
            return Ok(None);
        }
        let kind = store::log_kind(&shared.db(), JOURNAL_LOG_ID)?;
        if kind.as_deref() != Some(KIND_JOURNAL) {
            return Ok(None);
        }
        self.open_locked(state, JOURNAL_LOG_ID).map(Some)
    }

    pub(crate) fn open_locked(
        &self,
        state: &mut ShelfState,
        log_id: &str,
    ) -> Result<ShelfBook, ShelfError> {
        if state.released {
            return Err(ShelfError::Released);
        }
        if let Some(book) = state.books.get(log_id) {
            return Ok(book.clone());
        }
        let shared = self.shared();
        let kind = store::log_kind(&shared.db(), log_id)?;
        if state.hosts.contains_key(log_id) || kind.as_deref() == Some(KIND_HOST) {
            return Err(ShelfError::Invalid(format!(
                "{log_id} is a host log, not a book"
            )));
        }
        let kind = if kind.as_deref() == Some(KIND_JOURNAL) {
            KIND_JOURNAL
        } else {
            KIND_BOOK
        };
        let log = Log::open(&shared.path, log_id, &shared.checkpoint_key, &[], true)?;
        let indexes = shared
            .authenticated
            .as_ref()
            .map_or(Indexes::Defaults, |f| f(log_id));
        let book = ShelfBook::open(shared.clone(), log_id, kind, log, indexes)?;
        state.books.insert(log_id.to_string(), book.clone());
        Ok(book)
    }

    /// The ids of the books that hold records, in id order. The journal is
    /// not among them: it counts toward no completeness and no tally
    /// (evidencebook#461 §2.7).
    pub fn books(&self) -> Result<Vec<String>, ShelfError> {
        let state = self.lock();
        if state.released {
            return Err(ShelfError::Released);
        }
        let db = self.shared().db();
        let mut stmt = db.prepare(
            "SELECT DISTINCT book_id FROM book_records WHERE book_id NOT IN (SELECT log_id FROM shelf_logs WHERE kind = 'journal') ORDER BY book_id",
        )?;
        let ids = stmt.query_map([], |r| r.get(0))?;
        Ok(ids.collect::<Result<_, _>>()?)
    }

    /// Open an existing log of the shelf's file that is not a book and
    /// register it with the cadence (Go `Legacy`): it counts into the
    /// lifetime log count and is live until it has ended and a witnessed
    /// tick carries its final checkpoint. It never provisions a log. A
    /// book's id is refused.
    pub fn legacy(&self, log_id: &str) -> Result<HostLog, ShelfError> {
        let mut state = self.lock();
        self.host_locked(&mut state, log_id, true, false)
    }

    /// Open `log_id` as a host log, provisioning its CLL log when the file
    /// holds nothing of it (Go `HostLog`). With `in_cadence` the cadence
    /// checkpoints it like a book; otherwise the cadence never does, and the
    /// host checkpoints and witnesses it on its backend. It is the same host
    /// log for the same id for the shelf's lifetime. An id opened before in
    /// the other mode, in this process or as the file records it, or a
    /// book's id, is refused.
    pub fn host_log(&self, log_id: &str, in_cadence: bool) -> Result<HostLog, ShelfError> {
        let mut state = self.lock();
        self.host_locked(&mut state, log_id, in_cadence, true)
    }

    pub(crate) fn host_locked(
        &self,
        state: &mut ShelfState,
        log_id: &str,
        in_cadence: bool,
        provision: bool,
    ) -> Result<HostLog, ShelfError> {
        if state.released {
            return Err(ShelfError::Released);
        }
        if let Some(host) = state.hosts.get(log_id) {
            if host.in_cadence != in_cadence {
                return Err(ShelfError::Invalid(format!(
                    "host log {log_id} is already open {} the cadence",
                    if host.in_cadence { "in" } else { "outside" }
                )));
            }
            return Ok(HostLog {
                shelf: self.clone(),
                core: host.clone(),
            });
        }
        if state.books.contains_key(log_id) {
            return Err(ShelfError::Invalid(format!(
                "{log_id} is a book, not a host log"
            )));
        }
        if state.cadence.as_ref().is_some_and(|(id, _)| id == log_id) {
            return Err(ShelfError::Invalid(format!("{log_id} is the cadence log")));
        }
        let shared = self.shared();
        {
            let db = shared.db();
            let kind = store::log_kind(&db, log_id)?;
            if kind.as_deref() == Some(KIND_JOURNAL) {
                return Err(ShelfError::Invalid(format!(
                    "{log_id} is the shelf's journal, not a host log"
                )));
            }
            if kind.as_deref() == Some(KIND_BOOK) || store::record_count(&db, log_id)? > 0 {
                return Err(ShelfError::Invalid(format!(
                    "{log_id} is a book, not a host log"
                )));
            }
        }
        let log = Log::open(&shared.path, log_id, &shared.checkpoint_key, &[], provision)?;
        let claimed = (|| -> Result<(), ShelfError> {
            let mut db = shared.db();
            let tx = store::write_tx(&mut db)?;
            let held = store::log_kind(&tx, log_id)?;
            let known: i64 = tx.query_row(
                "SELECT COUNT(*) FROM cadence_logs WHERE log_id = ?",
                [log_id],
                |r| r.get(0),
            )?;
            // The mode is durable without a column of its own: a host log in
            // the cadence is in cadence_logs from the transaction that first
            // opens it (rows are never deleted), and one outside it never is.
            if !in_cadence && known > 0 {
                return Err(ShelfError::Invalid(format!(
                    "host log {log_id} is in the cadence"
                )));
            }
            if in_cadence && held.as_deref() == Some(KIND_HOST) && known == 0 {
                return Err(ShelfError::Invalid(format!(
                    "host log {log_id} is outside the cadence"
                )));
            }
            store::claim_kind(&tx, log_id, KIND_HOST)?;
            if in_cadence && known == 0 {
                store::mark_live(&tx, log_id)?;
            }
            tx.commit()?;
            Ok(())
        })();
        if let Err(e) = claimed {
            let _ = log.release();
            return Err(e);
        }
        let core = Arc::new(HostCore {
            id: log_id.to_string(),
            log: Arc::new(log),
            in_cadence,
        });
        state.hosts.insert(log_id.to_string(), core.clone());
        Ok(HostLog {
            shelf: self.clone(),
            core,
        })
    }

    /// Release every book and log the shelf opened, close its file and give
    /// up the writer lock. A book or host log from this shelf refuses writes
    /// afterwards. Idempotent.
    pub fn release(&self) -> Result<(), ShelfError> {
        let mut state = self.lock();
        if state.released {
            return Ok(());
        }
        state.released = true;
        self.shared().closed.store(true, Ordering::SeqCst);
        let mut first: Option<ShelfError> = None;
        let mut keep = |r: Result<(), ShelfError>| {
            if let Err(e) = r {
                first.get_or_insert(e);
            }
        };
        for book in state.books.values() {
            book.0.lock().released = true;
            keep(book.0.log.release());
        }
        for host in state.hosts.values() {
            keep(host.log.release());
        }
        if let Some((_, log)) = &state.cadence {
            keep(log.release());
        }
        // Close the shelf's own connection by replacing it with an
        // in-memory one: the file is closed before the lock is given up.
        {
            let mut db = self.shared().db();
            match Connection::open_in_memory() {
                Ok(empty) => {
                    let old = std::mem::replace(&mut *db, empty);
                    keep(old.close().map_err(|(_, e)| e.into()));
                }
                Err(e) => keep(Err(e.into())),
            }
        }
        if let Some(lock) = state.lock.take() {
            keep(lock.release());
        }
        match first {
            Some(e) => Err(e),
            None => Ok(()),
        }
    }
}

/// What a host log's writes run on (Go `HostLogStores`): the log's CLL
/// backend on the shelf's file and the shelf's artifact store, under the
/// shelf's one namespace. Both are the shelf's; they close with it. The
/// backend is the concrete SQLite store, so it coerces to whichever trait
/// object the host's log layer takes (an `Arc<dyn EntryStore + Send +
/// Sync>` for capsule-emit's ledger, an `Arc<dyn CheckpointLog>` for its
/// checkpoint state).
pub struct HostLogStores {
    pub log: Arc<cll::log::sqlite::SqliteStore>,
    pub originals: Arc<dyn ArtifactStore + Send + Sync>,
}

/// A log on the shelf whose records the host seals itself, in its own
/// record format, never converted into book records (evidencebook#254
/// §2.7; Go `HostLog`). `shelf_logs` records it as a host log, so
/// [`Shelf::open`] refuses its id across reopens and languages.
#[derive(Clone)]
pub struct HostLog {
    shelf: Shelf,
    core: Arc<HostCore>,
}

impl HostLog {
    /// The log's id.
    pub fn log_id(&self) -> &str {
        &self.core.id
    }

    /// Whether the cadence checkpoints it.
    pub fn in_cadence(&self) -> bool {
        self.core.in_cadence
    }

    /// Run `write` on the log's stores under the shelf's lock (Go `Write`).
    /// For a log in the cadence the log is put back in the live set first,
    /// so no write, finished or not, is left outside the next tick. `write`
    /// must not call the shelf or its cadence: the lock is held.
    pub fn write<T, E>(&self, write: impl FnOnce(&HostLogStores) -> Result<T, E>) -> Result<T, E>
    where
        E: From<ShelfError>,
    {
        let state = self.shelf.lock();
        if state.released {
            return Err(ShelfError::Released.into());
        }
        if self.core.in_cadence {
            let mut db = self.shelf.shared().db();
            let tx = store::write_tx(&mut db)?;
            store::mark_live(&tx, &self.core.id)?;
            tx.commit().map_err(ShelfError::from)?;
        }
        let _log = self.core.log.mu.lock().unwrap_or_else(|p| p.into_inner());
        let result = write(&self.stores_unchecked());
        drop(state);
        result
    }

    /// The log's stores, for a host log outside the cadence whose host keeps
    /// them open for as long as it runs (its own log layer and checkpoint
    /// state on them, under the host's own held lock). The cadence
    /// never touches such a log, so the host's writes need no shelf lock. A
    /// log in the cadence is refused: every write to it goes through
    /// [`HostLog::write`].
    pub fn stores(&self) -> Result<HostLogStores, ShelfError> {
        if self.core.in_cadence {
            return Err(ShelfError::Invalid(format!(
                "host log {} is in the cadence; write to it through HostLog::write",
                self.core.id
            )));
        }
        if self.shelf.lock().released {
            return Err(ShelfError::Released);
        }
        Ok(self.stores_unchecked())
    }

    fn stores_unchecked(&self) -> HostLogStores {
        HostLogStores {
            log: self.core.log.store.clone(),
            originals: self.shelf.shared().originals.clone(),
        }
    }
}