Skip to main content

borg_mesh/
sync.rs

1//! Two members of one collective reconcile its logs over iroh (QUIC, TLS with raw ed25519 keys).
2//!
3//! iroh authenticates the peer's key during the handshake; a node then checks that key against
4//! its verified membership before it sends a single byte of the collective. One session is a
5//! sequence of swaps on one stream (the dialler speaks first, the listener answers after it has
6//! checked what it heard). The order is the one PLAN.md "Reconnect" fixes, trust first:
7//!
8//! 1. `Hello`: which collective (the listener refuses a name that is not its own), and the port the
9//!    dialler listens on. The listener remembers it in peers.json at the IP QUIC saw the dialler
10//!    come from, so a node bound to 0.0.0.0 is still dialled back at an address that works.
11//! 2. `Membership`: both nodes' whole history of signed versions; each resolves it from its own
12//!    root (membership.rs: every step signed by an admin of the previous one, forks resolved the
13//!    same way everywhere). The peer must still be a member under the result, and revocations are
14//!    applied (quarantine) before any data moves, also when the held version did not change but
15//!    the logs were never checked against it (`origin::apply_if_new`).
16//! 3. `lessons.jsonl`, `stats.jsonl`, `outcomes-team.jsonl`, `playbooks.jsonl`, `constraints.jsonl`, `messages.jsonl`
17//!    (lines.rs `SYNCED` and the removed playbooks' step, `PLAYBOOKS`; a 0.3.0 peer's `borg/mesh/12`
18//!    session stops before the messages), each in turn:
19//!    range-based set reconciliation (recon.rs), cost ∝ difference: one fingerprint when nothing
20//!    changed. Then `Lines`: what the other side lacks, each line with its
21//!    signed origin (origin.rs). A line is stored only if the peer announced it, its origin verifies,
22//!    the membership admits its origin key at its stamp time, and it may travel under that key
23//!    (lines.rs `travels`: a message or seen line names its origin key as its node, and is fresh).
24//!    Messages also travel only within their `window` (borg msg.rs `within_windows`: what any sender's
25//!    window, msg.rs `window_for`, can have let through to one recipient, decided by no earlier line, so
26//!    pruning never takes a held line out of it), on offer and on receipt, and the log stays bounded as
27//!    `send` bounds it (msg.rs `bound`, bytes per member and class): a new line the bound would cut is
28//!    refused, so it is not taken again. What the log takes is decided before its lock and written
29//!    under it (`stored`, `Store::store_messages`), so a hook's acknowledgement never waits on it. A
30//!    `borg/mesh/13` peer is offered only what 0.5.0 takes (`v1_window`).
31//!    A pause, resume or veto from another node counts only once admitted under its node's role
32//!    (steer.rs `admit`), when it is stored.
33//!
34//!    The playbooks are removed, but their step is still spoken so a 0.3.0 or 0.4.0 peer keeps its
35//!    order: this node offers nothing there, follows the peer's reconciliation and stores nothing
36//!    (`skip`), as for a hive goal below.
37//!
38//!    3b. `Goals`: the removed hive's step, still spoken so a 0.3.0 peer keeps syncing on
39//!    `borg/mesh/12`. This node names no goal; for each valid goal id the peer names it follows the
40//!    peer's reconciliation of that goal's `LEGACY_GOAL_LOGS` logs and stores nothing (`skip`). What
41//!    a peer gave is remembered in memory (`Node::legacy`), so it is not sent again every session.
42//!
43//! One `Lines` frame carries at most `MAX_GIVE` bytes of lines; the rest go at the next session (the
44//! sets still differ, so its reconciliation finds them again).
45//! 4. `reasoning.jsonl` (git-note excerpts, notes in the borg crate): not reconciled with everyone.
46//!    Each author offers its own excerpts to the k = 2 members that rendezvous hashing assigns to
47//!    each excerpt (`replicas`), so the reasoning survives the author being offline without every
48//!    member holding all of it.
49//! 5. `Ask`: the commits whose reasoning a member wants (`borg notes fetch`); any node that holds
50//!    an admitted excerpt for one (or a link record with it on either side) answers with it, author
51//!    or replica, so the "why" can be read back while its author is offline.
52//! 6. `Live`: ephemeral, never reconciled into a log: presence (live.rs: which drones run where,
53//!    latest per member and session, a day at most). A 0.3.0 peer's hive beats and signed steering
54//!    directives in it are ignored: steering travels as messages, admitted when stored (steer.rs).
55//!
56//! A completed session, dialled or served, is recorded on each side by when it started, per member
57//! (`mesh/synced.json`, borg presence.rs `synced_path`), and by what its protocol says the member reads
58//! (`reads`, `mesh/reads.json`): `send` refuses a text the member's sessions could never be shown.
59//!
60//! Addressing is by key (addr.rs): a peer is dialled at its node id, with direct addresses and
61//! relay URLs as optional hints. How the endpoint reaches it across NAT is the collective's relay
62//! setting: n0's relays and address lookup (default), a self-hosted relay, or direct only.
63//!
64//! **Joining** (invite.rs) is a second ALPN: a node that is not a member yet presents a signed
65//! invite to the admin that issued it, and gets the membership history back (`join`, `admit`).
66
67use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet};
68use std::net::SocketAddr;
69use std::path::{Path, PathBuf};
70use std::sync::Mutex;
71use std::time::Duration;
72
73use borg::collective::{self, Config, Route};
74use borg::msg::{Line, V1_RATE_MS};
75use iroh::Endpoint;
76use iroh::endpoint::{Builder, ConnectOptions, Connection, Incoming, IncomingAddr, PortmapperConfig, RecvStream, SendStream, presets};
77use iroh::tls::CaTlsConfig;
78use serde::{Deserialize, Serialize};
79
80use crate::addr::{self, Hint, Relay, Target};
81use crate::identity::Identity;
82use crate::invite::{self, Invite, Ticket};
83use crate::lines::{self, Hash, REASONING, SYNCED, logs};
84use crate::membership::Membership;
85use crate::origin::{self, Origin};
86use crate::recon::{self, Recon};
87use crate::{from_hex, peers, to_hex};
88use crate::{live, steer};
89
90mod vectors;
91
92/// The session protocol. 14: messages v2 (borg msg.rs: texts kept whole up to `WIRE_TEXT`, `seen`
93/// lines, the acknowledgement window instead of 0.5.0's rate). 13: `messages.jsonl` joined the synced
94/// logs (lines.rs `SYNCED`). The logs reconcile in a fixed list, so a peer on another list would
95/// reconcile another file at the same step; 11 added `constraints.jsonl` and refused 10 at the
96/// handshake for that reason.
97pub const ALPN: &[u8] = b"borg/mesh/14";
98/// 0.4.0 and 0.5.x's protocol, still spoken whole: `ALPN`'s steps, but such a peer reads only v1
99/// message lines and keeps only what 0.5.0's window takes (`V1_RATE_MS`, `V1_KEEP`), so it is offered
100/// only that (`v1_window`): no v2 text, no seen line, one tell or ask per sender and recipient per
101/// rate slot.
102pub const V13_ALPN: &[u8] = b"borg/mesh/13";
103/// 0.3.0's protocol, still spoken whole: the session of `ALPN` without the messages (`synced`). 12
104/// already outlived the hive: its steps (`Goals`, the beats of `Live`) are spoken empty.
105pub const LEGACY_ALPN: &[u8] = b"borg/mesh/12";
106/// The protocols of older nodes (`borg/mesh/11` down to `/1`), offered and accepted beside `ALPN` only to
107/// tell a mismatch: without them the handshake fails and nothing says why. A session on one is closed at
108/// once and the peer recorded (`mismatch`). A newer node does the same with this one's `ALPN`.
109fn older() -> Vec<Vec<u8>> {
110    let below = version(LEGACY_ALPN).unwrap_or(0);
111    (1..below).rev().map(|v| format!("borg/mesh/{v}").into_bytes()).collect()
112}
113
114/// The other side of a session, as the handshake left it: its member key and the protocol it speaks.
115#[derive(Clone, Copy)]
116struct Peer<'a> {
117    key: &'a str,
118    alpn: &'a [u8],
119}
120
121/// Whether a session on `alpn` syncs with this node: `ALPN`, a 0.4.0 or 0.5.x peer's `V13_ALPN`, or a
122/// 0.3.0 peer's `LEGACY_ALPN`.
123fn spoken(alpn: &[u8]) -> bool {
124    alpn == ALPN || alpn == V13_ALPN || alpn == LEGACY_ALPN
125}
126
127/// The removed playbooks' log. Its step (after the outcomes) stays in the session, so `ALPN` and
128/// `LEGACY_ALPN` keep their order; a peer that still reconciles it is followed and nothing is kept
129/// (`skip`). A leftover file in a store is neither offered nor stamped.
130const PLAYBOOKS: &str = "playbooks.jsonl";
131
132/// The logs a session on `ALPN` steps through, in order: `SYNCED` with the `PLAYBOOKS` step.
133const STEPS: [&str; 6] = [SYNCED[0], SYNCED[1], SYNCED[2], PLAYBOOKS, SYNCED[3], SYNCED[4]];
134
135/// The logs a session on `alpn` steps through, in order: all of `STEPS`, or on `LEGACY_ALPN` the five a
136/// 0.3.0 peer knows.
137fn synced(alpn: &[u8]) -> impl Iterator<Item = &'static str> {
138    let legacy = alpn == LEGACY_ALPN;
139    STEPS.into_iter().filter(move |f| !legacy || *f != borg::msg::FILE)
140}
141
142/// The number of a session protocol (`borg/mesh/<n>`).
143fn version(alpn: &[u8]) -> Option<u32> {
144    std::str::from_utf8(alpn).ok()?.strip_prefix("borg/mesh/")?.parse().ok()
145}
146/// Protocol: the newest message line format (borg msg.rs `Message::v`) a peer whose session ran on
147/// `alpn` reads: 2 on `ALPN`; 1 on `V13_ALPN`, whose 0.5.0 drops a v2 line; 0 on `LEGACY_ALPN`, which
148/// syncs no messages (`synced`).
149fn reads(alpn: &[u8]) -> u8 {
150    if alpn == ALPN {
151        2
152    } else if alpn == V13_ALPN {
153        1
154    } else {
155        0
156    }
157}
158/// Protocol: the join handshake, an invite for the membership history (`Node::join`, `Node::admit`).
159pub const JOIN_ALPN: &[u8] = b"borg/join/1";
160/// Guard: the largest frame accepted; a lessons log is far smaller, a bigger frame is a broken or
161/// hostile peer, which would otherwise make this node allocate what it claims.
162const MAX_FRAME: usize = 256 << 20;
163/// Guard: the most bytes of lines one `Lines` frame gives (well within `MAX_FRAME`): a big difference
164/// goes over several sessions, never in a frame the peer refuses.
165const MAX_GIVE: usize = 64 << 20;
166/// Baseline: how many members hold a replica of each reasoning excerpt besides its author (two, so an
167/// excerpt outlives one member going away; design-chosen).
168pub const REPLICAS: usize = 2;
169/// Guard: the most commits one `Ask` names, and most excerpts one answer carries, so one round of a
170/// long history cannot make a peer read every excerpt it holds.
171const MAX_ASK: usize = 1000;
172/// Protocol: how many logs per goal a 0.3.0 peer reconciles in step 3b (its hive's synced logs: goal, tasks,
173/// contracts, messages, flags, bundles, spend).
174const LEGACY_GOAL_LOGS: usize = 7;
175// 0.5.0's message pacing is borg msg.rs `V1_RATE_MS` (protocol: its `RATE_MS`), which a `V13_ALPN` peer
176// applies to what it holds and receives: one tell or ask per sender and recipient that often. It is all
177// such a peer takes, so it is all it is offered (`v1_window`); this node's `send` paces by msg.rs
178// `window_for`, and the mesh bounds by msg.rs `within_windows`.
179/// Protocol: 0.5.0's line bound (its msg.rs `KEEP`), which a `V13_ALPN` peer shares equally among the
180/// members (`v1_share`) and applies on receipt, as `V1_RATE_MS`.
181const V1_KEEP: usize = 500;
182
183/// Where a node finds its collective. Kept instead of a fixed `Route` so every session re-reads the
184/// config: setting the collective to `share: local`, adding the repo to `off` or breaking the file
185/// stops a running `serve` at its next session, not at its next restart.
186#[derive(Debug, Clone)]
187pub struct Place {
188    /// The collectives config (`BORG_CONFIG`).
189    pub config: PathBuf,
190    /// The root store (`BORG_HOME`).
191    pub base: PathBuf,
192    /// The directory whose collective this is (`--dir`).
193    pub dir: String,
194    /// The collective named directly (`--collective`, and `join` before any repo routes to it):
195    /// it must be listed in the config, and its `share` still decides.
196    pub collective: Option<String>,
197}
198
199impl Place {
200    pub fn from_env(dir: &str) -> Place {
201        Place { config: collective::config_path(), base: borg::learn::home(), dir: dir.into(), collective: None }
202    }
203
204    /// The same machine, the collective named `name` instead of the one a directory routes to.
205    pub fn named(&self, name: &str) -> Place {
206        Place { collective: Some(name.into()), ..self.clone() }
207    }
208
209    /// The route as of now; a broken config never leaks a repo into a collective, it routes off.
210    pub fn route(&self) -> Route {
211        self.resolve().1
212    }
213
214    fn resolve(&self) -> (Option<Config>, Route) {
215        match collective::load_from(&self.config) {
216            Ok(cfg) => {
217                let r = match &self.collective {
218                    None => collective::route(cfg.as_ref(), &self.dir, &self.base),
219                    Some(name) => match cfg.as_ref() {
220                        None if name == "default" => collective::route(None, &self.dir, &self.base),
221                        None => Route::Off,
222                        Some(c) => c.collectives.iter().find(|e| e.name == *name).map_or(Route::Off, |e| Route::Collective {
223                            name: e.name.clone(),
224                            home: collective::store(&self.base, &e.name),
225                            share: e.share,
226                        }),
227                    },
228                };
229                (cfg, r)
230            }
231            Err(_) => (None, Route::Off),
232        }
233    }
234
235    /// How this collective reaches its members (addr.rs): `BORG_RELAY`, else its `relay` setting.
236    pub fn relay(&self) -> Result<Relay, String> {
237        let (cfg, route) = self.resolve();
238        let configured = match (&cfg, &route) {
239            (Some(c), Route::Collective { name, .. }) => c.collectives.iter().find(|e| e.name == *name).and_then(|e| e.relay.clone()),
240            _ => None,
241        };
242        Relay::choose(configured.as_deref())
243    }
244
245    /// The gate on the config as it is now: the collective's name and store, or why nothing may sync.
246    pub fn gate(&self) -> Result<(String, PathBuf), String> {
247        let (cfg, route) = self.resolve();
248        if let (Some(name), Route::Off) = (&self.collective, &route) {
249            return Err(format!("collective {name:?} is not in {}", self.config.display()));
250        }
251        let (name, home) = gate(&route)?;
252        if let Some(local) = collective::local_owner_of(cfg.as_ref(), home, &self.base) {
253            return Err(format!("store {} also belongs to local collective {local:?}: refusing to sync", home.display()));
254        }
255        Ok((name.to_string(), home.to_path_buf()))
256    }
257}
258
259/// The gate. A collective's data may leave this machine only if `collective::may_sync` says so.
260pub fn gate(route: &Route) -> Result<(&str, &Path), String> {
261    match route {
262        Route::Collective { name, home, .. } if collective::may_sync(route) => Ok((name, home)),
263        Route::Collective { name, .. } => Err(format!("collective {name:?} is local: it never leaves this machine, refusing to sync")),
264        Route::Off => Err("this repo belongs to no collective (off): nothing may sync".into()),
265    }
266}
267
268/// What one session moved.
269#[derive(Debug, Default, Clone, Copy, PartialEq, Eq)]
270pub struct Report {
271    /// Lines this node sent / newly stored (lessons and reasoning).
272    pub sent: usize,
273    pub received: usize,
274    /// Lines the peer sent that were refused (bad origin, or an origin the membership no longer admits).
275    pub rejected: usize,
276    /// Lines moved to quarantine because a newer membership revoked their origin.
277    pub quarantined: usize,
278    /// Reconciliation of lessons: rounds, fingerprints and hashes this node sent.
279    pub rounds: usize,
280    pub fps: usize,
281    pub hashes: usize,
282    /// Reasoning excerpts this node handed to the peer as one of their replicas.
283    pub replicated: usize,
284    /// Reasoning excerpts this node answered an `Ask` with.
285    pub answered: usize,
286    /// Remote presence records that were new or newer (live.rs).
287    pub presence: usize,
288    /// Another node's pauses, resumes and vetoes newly admitted / refused (steer.rs).
289    pub steered: usize,
290    pub dropped: usize,
291    /// Members this node admitted with an invite (`admit`).
292    pub joined: usize,
293}
294
295/// A line and where it came from.
296#[derive(Debug, Clone, Serialize, Deserialize)]
297pub struct Signed {
298    pub line: String,
299    pub origin: Origin,
300}
301
302#[derive(Debug, Serialize, Deserialize)]
303#[serde(tag = "t", rename_all = "snake_case")]
304enum Msg {
305    /// `relay`: the dialler's home relay, so the listener can dial it back across NAT (absent from
306    /// older nodes).
307    Hello {
308        collective: String,
309        listen: Option<String>,
310        #[serde(default)]
311        relay: Option<String>,
312    },
313    Refused {
314        reason: String,
315    },
316    Join {
317        invite: Invite,
318        listen: Option<String>,
319        relay: Option<String>,
320    },
321    Welcome {
322        docs: Vec<Membership>,
323    },
324    Membership {
325        docs: Vec<Membership>,
326    },
327    Round {
328        round: recon::Round,
329    },
330    Lines {
331        lines: Vec<Signed>,
332    },
333    /// The hive goals a side holds (3b): none from this node, a 0.3.0 peer's are skipped.
334    Goals {
335        ids: Vec<String>,
336    },
337    Offer {
338        hashes: Vec<String>,
339    },
340    Want {
341        hashes: Vec<String>,
342    },
343    Ask {
344        commits: Vec<String>,
345    },
346    Live {
347        presence: Vec<borg::presence::Remote>,
348        #[serde(default)]
349        directives: Directives,
350    },
351}
352
353/// 0.3.0's signed steering directives in `Live`: sent empty, as a 0.3.0 peer requires the field, and
354/// a 0.3.0 peer's ignored unread. Steering travels as messages now (steer.rs).
355#[derive(Debug, Default)]
356struct Directives;
357
358impl Serialize for Directives {
359    fn serialize<S: serde::Serializer>(&self, s: S) -> Result<S::Ok, S::Error> {
360        s.collect_seq(std::iter::empty::<()>())
361    }
362}
363
364impl<'de> Deserialize<'de> for Directives {
365    fn deserialize<D: serde::Deserializer<'de>>(d: D) -> Result<Self, D::Error> {
366        serde::de::IgnoredAny::deserialize(d).map(|_| Directives)
367    }
368}
369
370/// Which CAs vouch for a relay's, address lookup's or DNS-over-HTTPS server's certificate: the
371/// system store, so a company proxy that re-signs TLS with its own CA (installed on the machine)
372/// does not cut the node off its relay. A machine with no readable store falls back to Mozilla's.
373/// Peer connections never depend on this: iroh checks those by key.
374fn ca_trust() -> CaTlsConfig {
375    let system = CaTlsConfig::system();
376    match system.server_cert_verifier(iroh::tls::default_provider()) {
377        Ok(_) => system,
378        Err(_) => CaTlsConfig::embedded(),
379    }
380}
381
382/// One node of one collective: its key, its store and an iroh endpoint.
383pub struct Node {
384    place: Place,
385    /// The store the node was bound to; a session whose route now names another store is refused.
386    home: PathBuf,
387    me: Identity,
388    endpoint: Endpoint,
389    /// Serialises this node's writes to its store (sessions run concurrently under `serve`).
390    store: Mutex<()>,
391    /// Whether peers are told where this node listens (false for a one-shot dial).
392    announce: bool,
393    /// How peers are reached (addr.rs), fixed when the endpoint was built.
394    relay: Relay,
395    /// The hashes each peer gave per log this node follows and keeps nowhere (`skip`: the playbooks, or
396    /// a 0.3.0 peer's hive goal log, 3b), by (peer, goal or file, log): never stored, so the next dial
397    /// reconciles against them and the peer gives only what is new.
398    legacy: Mutex<HashMap<(String, String, usize), BTreeSet<Hash>>>,
399}
400
401impl Node {
402    /// Opens the endpoint for `place` on `addr` — only after the gate, so a local collective or an
403    /// off repo never even gets a socket.
404    pub async fn bind(place: &Place, addr: SocketAddr) -> Result<Node, String> {
405        Node::open(place, Some(addr), true, |b| b).await
406    }
407
408    /// A node of a collective whose membership this machine does not hold yet: it can only `join`.
409    pub async fn joining(place: &Place, addr: SocketAddr) -> Result<Node, String> {
410        Node::open(place, Some(addr), false, |b| b).await
411    }
412
413    /// The endpoint for the collective's relay setting (addr.rs). `addr` is the UDP socket for
414    /// direct paths (`None`: relay only, for tests); `member`: refuse without a membership;
415    /// `tweak` adjusts the builder (tests trust a local relay's self-signed certificate).
416    pub async fn open(place: &Place, addr: Option<SocketAddr>, member: bool, tweak: impl FnOnce(Builder) -> Builder) -> Result<Node, String> {
417        let (_, home) = place.gate()?;
418        let relay = place.relay()?;
419        let me = Identity::load_or_create(&home)?;
420        if member {
421            members(&home)?;
422        }
423        let builder = match &relay {
424            // n0's relays and DNS address lookup (pkarr): found by key alone, hole-punched across NAT.
425            Relay::Default => Endpoint::builder(presets::N0),
426            // A self-hosted relay and no public lookup: nothing about this node reaches n0.
427            Relay::Custom(_) => Endpoint::builder(presets::Minimal).relay_mode(relay.mode()),
428            // Direct addresses only: no relay, no lookup, no probing of the LAN's router.
429            Relay::Disabled => Endpoint::builder(presets::Minimal).relay_mode(relay.mode()).portmapper_config(PortmapperConfig::Disabled),
430        };
431        let mut builder = builder
432            .secret_key(me.iroh_secret())
433            .alpns([vec![ALPN.to_vec(), JOIN_ALPN.to_vec(), V13_ALPN.to_vec(), LEGACY_ALPN.to_vec()], older()].concat())
434            .clear_ip_transports()
435            .ca_tls_config(ca_trust());
436        if let Some(a) = addr {
437            builder = builder.bind_addr(a).map_err(|e| e.to_string())?;
438        }
439        let endpoint = tweak(builder).bind().await.map_err(|e| format!("bind {}: {e}", addr.map(|a| a.to_string()).unwrap_or("relay".into())))?;
440        Ok(Node { place: place.clone(), home, me, endpoint, store: Mutex::new(()), announce: true, relay, legacy: Mutex::default() })
441    }
442
443    /// Waits until the endpoint holds a connection to its home relay (at most `within`); true if
444    /// it does. Nothing to wait for with relays disabled.
445    pub async fn online(&self, within: Duration) -> bool {
446        self.relay == Relay::Disabled || tokio::time::timeout(within, self.endpoint.online()).await.is_ok()
447    }
448
449    /// The relay this node is reachable through, once it is online.
450    pub fn home_relay(&self) -> Option<String> {
451        self.endpoint.addr().relay_urls().next().map(|u| u.to_string())
452    }
453
454    /// Where another machine could reach this node: its direct addresses (`tickets`) and its home
455    /// relay. What `serve` writes to `mesh/addr.json` for `invite` to put in a ticket.
456    pub fn hints(&self) -> Vec<Hint> {
457        let mut h: Vec<Hint> = self.tickets().iter().filter_map(|t| Target::parse(t).ok()).flat_map(|t| t.hints).collect();
458        h.extend(self.home_relay().and_then(|u| addr::hints(&u).ok()).into_iter().flatten());
459        h
460    }
461
462    pub fn relay(&self) -> &Relay {
463        &self.relay
464    }
465
466    /// A node that only dials and exits (`sync`, `fetch`): it tells no peer where it listens,
467    /// because its ephemeral port is gone a moment later.
468    pub fn quiet(mut self) -> Node {
469        self.announce = false;
470        self
471    }
472
473    /// The gate again, on the config as it is now, and the store must still be the bound one.
474    pub fn gate(&self) -> Result<(String, PathBuf), String> {
475        let (name, home) = self.place.gate()?;
476        if home != self.home {
477            return Err(format!("{} now routes to store {}, this node serves {}: restart it", self.place.dir, home.display(), self.home.display()));
478        }
479        Ok((name, home))
480    }
481
482    pub fn home(&self) -> &Path {
483        &self.home
484    }
485
486    pub fn public(&self) -> String {
487        self.me.public()
488    }
489
490    fn addr(&self) -> Option<SocketAddr> {
491        self.endpoint.bound_sockets().first().copied()
492    }
493
494    /// `<key>@<ip:port>`: what a peer passes to `sync`. The first of `tickets`.
495    pub fn ticket(&self) -> String {
496        self.tickets().into_iter().next().unwrap_or_else(|| format!("{}@{}", self.me.public(), self.addr().map(|a| a.to_string()).unwrap_or_default()))
497    }
498
499    /// One ticket per address another machine could dial. Bound to a specific IP that is the one;
500    /// bound to an unspecified IP (0.0.0.0, the default) it is each interface address iroh found,
501    /// never 0.0.0.0 itself (empty if there is none: then `--bind IP:PORT`).
502    pub fn tickets(&self) -> Vec<String> {
503        let Some(bound) = self.addr() else { return Vec::new() };
504        let addrs: Vec<SocketAddr> = if bound.ip().is_unspecified() {
505            let mut v: Vec<SocketAddr> = self
506                .endpoint
507                .addr()
508                .ip_addrs()
509                .filter(|a| !a.ip().is_unspecified() && a.is_ipv4() == bound.is_ipv4())
510                .map(|a| SocketAddr::new(a.ip(), bound.port()))
511                .collect();
512            v.sort_by_key(|a| (a.ip().is_loopback(), *a));
513            v.dedup();
514            v
515        } else {
516            vec![bound]
517        };
518        addrs.into_iter().map(|a| format!("{}@{a}", self.me.public())).collect()
519    }
520
521    /// What this node announces in its hello: the address it listens on (the port is what counts).
522    fn listen(&self) -> Option<String> {
523        self.announce.then(|| self.addr()).flatten().map(|a| a.to_string())
524    }
525
526    /// Applies the held membership to the logs unless they were already checked against it (a
527    /// `join` or `revoke` by another process, or a sync, changed it). Returns how many lines moved.
528    pub fn enforce(&self) -> Result<usize, String> {
529        let _w = self.store.lock().unwrap_or_else(|e| e.into_inner());
530        let m = members(&self.home)?;
531        steer::enforce(&self.home, &m, borg::learn::now_ms());
532        origin::apply_if_new(&self.home, &m)
533    }
534
535    /// Stamps the lines this node's hooks wrote since the last call as its own (`Store::stamp`).
536    pub fn stamp(&self) -> Result<usize, String> {
537        self.local().stamp()
538    }
539
540    /// This node's store, as a merge writes it (`Store`).
541    fn local(&self) -> Store<'_> {
542        Store { home: &self.home, lock: &self.store, me: &self.me }
543    }
544
545    /// Records a completed session with `remote` on `alpn` that started at `started` (borg presence.rs
546    /// `synced_path`): a claim written before it reached the peer, and the peer's earlier claims came
547    /// back. And what that node reads (`reads`, presence.rs `reads_path`), so `send` refuses a message
548    /// its sessions could never be shown.
549    fn synced(&self, remote: &str, alpn: &[u8], started: i64) {
550        let _w = self.store.lock().unwrap_or_else(|e| e.into_inner());
551        let _ = borg::presence::note_synced(&self.home, remote, started);
552        let _ = borg::presence::forget_mismatch(&self.home, remote);
553        let _ = borg::presence::note_reads(&self.home, remote, reads(alpn));
554    }
555
556    /// A session with member `remote` that came up on another protocol (`older`): recorded for `borg
557    /// doctor` (borg presence.rs `protocols_path`), closed, and the error says who has to upgrade; the
558    /// close reason says the same from the peer's side.
559    fn mismatch(&self, conn: &Connection, remote: &str) -> String {
560        let m = borg::presence::Mismatch {
561            theirs: String::from_utf8_lossy(conn.alpn()).into_owned(),
562            ours: String::from_utf8_lossy(ALPN).into_owned(),
563            ms: borg::learn::now_ms(),
564        };
565        let (line, told) = (m.line(remote), m.told(&self.me.public()));
566        {
567            let _w = self.store.lock().unwrap_or_else(|e| e.into_inner());
568            let _ = borg::presence::note_mismatch(&self.home, remote, m);
569        }
570        conn.close(2u32.into(), told.as_bytes());
571        line
572    }
573
574    /// Forgets `remote`'s last completed session (presence.rs `forget_synced`): a dial to it failed.
575    pub(crate) fn unreachable(&self, remote: &str) {
576        let _w = self.store.lock().unwrap_or_else(|e| e.into_inner());
577        let _ = borg::presence::forget_synced(&self.home, remote);
578    }
579
580    /// The home relay this node tells peers about (not for a one-shot dial, nor with relays off).
581    fn announced(&self) -> Option<String> {
582        self.announce.then(|| self.home_relay()).flatten()
583    }
584
585    /// Dials the member named by `target` (`<key>[@<hints>]`, addr.rs) and reconciles.
586    pub async fn sync(&self, target: &str) -> Result<Report, String> {
587        self.fetch(target, &[]).await
588    }
589
590    /// `sync`, also asking the peer for the reasoning excerpts of `commits` (step 5): what it
591    /// holds lands in this node's reasoning.jsonl.
592    pub async fn fetch(&self, target: &str, commits: &[String]) -> Result<Report, String> {
593        self.dial(target, commits, ALPN, [vec![V13_ALPN.to_vec(), LEGACY_ALPN.to_vec()], older()].concat()).await
594    }
595
596    /// `fetch` on the protocol `alpn`, with `also` offered beside it (a test speaks an older one alone).
597    async fn dial(&self, target: &str, commits: &[String], alpn: &[u8], also: Vec<Vec<u8>>) -> Result<Report, String> {
598        let (name, home) = self.gate()?;
599        let (name, home) = (name.as_str(), home.as_path());
600        let membership = members(home)?;
601        let t = Target::parse(target)?;
602        if membership.role_of(&t.key).is_none() {
603            return Err(format!("{} is not a member of {name}: refusing to connect", t.key));
604        }
605        let started = borg::learn::now_ms();
606        let connecting = self.endpoint.connect_with_opts(t.endpoint_addr(&self.relay)?, alpn, ConnectOptions::new().with_additional_alpns(also));
607        let conn = connecting.await.map_err(|e| format!("connect {target}: {e}"))?.await.map_err(|e| format!("connect {target}: {e}"))?;
608        if !spoken(conn.alpn()) {
609            return Err(self.mismatch(&conn, &t.key));
610        }
611        let result = async {
612            let remote = authorise(&conn, &membership, name)?;
613            let (mut tx, mut rx) = conn.open_bi().await.map_err(|e| e.to_string())?;
614            let hello = Msg::Hello { collective: name.into(), listen: self.listen(), relay: self.announced() };
615            let Msg::Hello { collective, .. } = swap(true, &mut tx, &mut rx, hello).await? else {
616                return Err("protocol: expected hello".into());
617            };
618            if collective != name {
619                return Err(format!("peer answered for collective {collective:?}, not {name:?}"));
620            }
621            let report = self.exchange(Peer { key: &remote, alpn: conn.alpn() }, true, &mut tx, &mut rx, membership, commits).await?;
622            let _ = tx.finish();
623            self.synced(&remote, conn.alpn(), started);
624            Ok(report)
625        }
626        .await;
627        conn.close(0u32.into(), if result.is_ok() { b"done" } else { b"error" });
628        vectors::reindexed(&self.home, result).await
629    }
630
631    /// Joins with an invite ticket (invite.rs): dials the admin that issued it, presents the signed
632    /// invite, installs the membership history it answers with (trusted because iroh authenticated
633    /// the ticket's admin key), remembers the admin as a peer and syncs with it at once.
634    pub async fn join(&self, ticket: &Ticket) -> Result<(Membership, Report), String> {
635        let (name, home) = self.gate()?;
636        let (name, home) = (name.as_str(), home.as_path());
637        if ticket.invite.collective != name {
638            return Err(format!("the ticket is for collective {:?}, this node is {name:?}", ticket.invite.collective));
639        }
640        let admin = ticket.target();
641        let conn = self.endpoint.connect(admin.endpoint_addr(&self.relay)?, JOIN_ALPN).await.map_err(|e| format!("connect {}: {e}", admin.key))?;
642        let me = self.public();
643        let result = async {
644            let (mut tx, mut rx) = conn.open_bi().await.map_err(|e| e.to_string())?;
645            let ask = Msg::Join { invite: ticket.invite.clone(), listen: self.listen(), relay: self.announced() };
646            let Msg::Welcome { docs } = swap(true, &mut tx, &mut rx, ask).await? else { return Err("protocol: expected welcome".into()) };
647            let _ = tx.finish();
648            let _w = self.store.lock().unwrap_or_else(|e| e.into_inner());
649            let installed = match origin::join(home, name, &docs) {
650                Ok((m, _)) => m,
651                // Joined before (a lost answer, a second ticket): the held version already lists us.
652                Err(e) => Membership::load(home)?.filter(|m| m.role_of(&me).is_some()).ok_or(e)?,
653            };
654            if installed.role_of(&me).is_none() {
655                return Err(format!("the admin's membership of {name} does not list this node ({me})"));
656            }
657            steer::enforce(home, &installed, borg::learn::now_ms());
658            Ok(installed)
659        }
660        .await;
661        conn.close(0u32.into(), if result.is_ok() { b"done" } else { b"error" });
662        let installed = result?;
663        peers::remember(home, &admin.key, &addr::join_hints(&admin.hints))?;
664        let report = self.sync(&admin.to_string()).await.map_err(|e| format!("joined {name}, but the first sync failed: {e}"))?;
665        Ok((installed, report))
666    }
667
668    /// Answers a node presenting an invite (`join`): checks it (invite.rs `redeem`: issued by this
669    /// node, signed, not expired, not spent, not withdrawn), adds the key with the invite's role,
670    /// and hands over the membership history.
671    async fn admit(&self, conn: &Connection, observed: Option<SocketAddr>) -> Result<Report, String> {
672        let (name, home) = self.gate()?;
673        let (name, home) = (name.as_str(), home.as_path());
674        let joiner = to_hex(conn.remote_id().as_bytes());
675        let (mut tx, mut rx) = conn.accept_bi().await.map_err(|e| e.to_string())?;
676        let Msg::Join { invite, listen, relay } = recv(&mut rx).await? else { return Err("protocol: expected join".into()) };
677        let verdict = {
678            let _w = self.store.lock().unwrap_or_else(|e| e.into_inner());
679            invite::redeem(home, &self.me, name, &invite, &joiner, borg::learn::now_ms())
680        };
681        let m = match verdict {
682            Ok(m) => m,
683            Err(reason) => {
684                send(&mut tx, &Msg::Refused { reason: reason.clone() }).await?;
685                let _ = tx.finish();
686                // Let the joiner read why before the connection goes (it closes once it has).
687                let _ = tokio::time::timeout(Duration::from_secs(5), conn.closed()).await;
688                return Err(format!("join by {joiner}: {reason}"));
689            }
690        };
691        steer::enforce(home, &m, borg::learn::now_ms());
692        heard(home, &joiner, listen, relay, observed);
693        send(&mut tx, &Msg::Welcome { docs: Membership::history(home) }).await?;
694        let _ = tx.finish();
695        conn.closed().await; // the joiner closes once it has read everything
696        Ok(Report { joined: 1, ..Report::default() })
697    }
698
699    /// The next peer dialling in, `None` once the endpoint is closed.
700    pub async fn incoming(&self) -> Option<Incoming> {
701        self.endpoint.accept().await
702    }
703
704    /// Serves the next peer that dials in; returns what it moved, or why it was turned away.
705    pub async fn accept(&self) -> Result<Report, String> {
706        self.handle(self.incoming().await.ok_or("endpoint closed")?).await
707    }
708
709    /// Serves one peer that dialled in.
710    pub async fn handle(&self, incoming: Incoming) -> Result<Report, String> {
711        // The IP the dialler's packets came from: where it can be reached, whatever it bound to.
712        let observed = match incoming.remote_addr() {
713            IncomingAddr::Ip(a) => Some(a),
714            _ => None,
715        };
716        let started = borg::learn::now_ms();
717        let conn = incoming.await.map_err(|e| format!("handshake: {e}"))?;
718        if !spoken(conn.alpn()) && conn.alpn() != JOIN_ALPN {
719            let (name, home) = self.gate()?;
720            let remote = authorise(&conn, &members(&home)?, &name)?;
721            return Err(self.mismatch(&conn, &remote));
722        }
723        let session = async {
724            let (name, home) = self.gate()?;
725            let (name, home) = (name.as_str(), home.as_path());
726            let membership = members(home)?;
727            let remote = authorise(&conn, &membership, name)?;
728            let (mut tx, mut rx) = conn.accept_bi().await.map_err(|e| e.to_string())?;
729            let Msg::Hello { collective, listen, relay } = recv(&mut rx).await? else { return Err("protocol: expected hello".into()) };
730            if collective != name {
731                let reason = format!("this node serves collective {name:?}, not {collective:?}");
732                send(&mut tx, &Msg::Refused { reason: reason.clone() }).await?;
733                return Err(reason);
734            }
735            heard(home, &remote, listen, relay, observed);
736            send(&mut tx, &Msg::Hello { collective: name.into(), listen: self.listen(), relay: self.announced() }).await?;
737            let report = self.exchange(Peer { key: &remote, alpn: conn.alpn() }, false, &mut tx, &mut rx, membership, &[]).await?;
738            let _ = tx.finish();
739            conn.closed().await; // the dialler closes once it has read everything
740            self.synced(&remote, conn.alpn(), started);
741            Ok(report)
742        };
743        let result = if conn.alpn() == JOIN_ALPN { self.admit(&conn, observed).await } else { session.await };
744        if let Err(e) = &result {
745            conn.close(1u32.into(), e.as_bytes());
746        }
747        vectors::reindexed(&self.home, result).await
748    }
749
750    pub async fn close(&self) {
751        self.endpoint.close().await;
752    }
753
754    /// The session with `peer` after both sides know who they talk to (module doc, steps 2–6).
755    async fn exchange(
756        &self,
757        peer: Peer<'_>,
758        dialler: bool,
759        tx: &mut SendStream,
760        rx: &mut RecvStream,
761        mine: Membership,
762        ask: &[String],
763    ) -> Result<Report, String> {
764        let Peer { key: remote, alpn } = peer;
765        let home = self.home.as_path();
766        let mut report = Report::default();
767        let Msg::Membership { docs } = swap(dialler, tx, rx, Msg::Membership { docs: Membership::history(home) }).await? else {
768            return Err("protocol: expected membership".into());
769        };
770        let mine = {
771            let _w = self.store.lock().unwrap_or_else(|e| e.into_inner());
772            let (held, _) = Membership::adopt(home, &docs).unwrap_or((mine, false));
773            report.quarantined += origin::apply_if_new(home, &held)?;
774            steer::enforce(home, &held, borg::learn::now_ms()); // a node that lost the right to steer, or an expired steer, stops here
775            held
776        };
777        if mine.role_of(remote).is_none() {
778            return Err(format!("{remote} is no longer a member of {}", mine.collective));
779        }
780        self.stamp()?;
781        let origins = origin::load(home);
782
783        // Lessons, and the other synced logs: reconcile with everyone. The playbooks: followed, nothing kept.
784        for file in synced(alpn) {
785            if file == PLAYBOOKS {
786                self.skip(dialler, tx, rx, (remote.to_string(), file.to_string(), 0)).await?;
787            } else {
788                self.reconcile(dialler, tx, rx, &home.join(file), &origins, &mine, alpn == V13_ALPN, &mut report).await?;
789            }
790        }
791
792        // A 0.3.0 peer's hive goals (3b): followed, nothing kept.
793        let Msg::Goals { ids } = swap(dialler, tx, rx, Msg::Goals { ids: Vec::new() }).await? else {
794            return Err("protocol: expected goals".into());
795        };
796        for goal in legacy_goals(ids) {
797            for log in 0..LEGACY_GOAL_LOGS {
798                self.skip(dialler, tx, rx, (remote.to_string(), goal.clone(), log)).await?;
799            }
800        }
801
802        // Reasoning: each author hands its excerpts to their replicas.
803        let path = home.join(REASONING);
804        let held = lines::read(&path);
805        let set = origin::admitted(held.clone(), &origins, &mine);
806        self.gate()?;
807        let me = self.me.public();
808        let offer: BTreeSet<Hash> =
809            set.keys().filter(|h| origins.get(*h).is_some_and(|o| o.key == me) && replicas(h, &mine, &me).iter().any(|k| k == remote)).copied().collect();
810        let Msg::Offer { hashes } = swap(dialler, tx, rx, Msg::Offer { hashes: offer.iter().map(|h| to_hex(h)).collect() }).await? else {
811            return Err("protocol: expected offer".into());
812        };
813        let offered: BTreeSet<Hash> = hashes.iter().map(|h| from_hex::<32>(h)).collect::<Result<_, _>>()?;
814        let want: BTreeSet<Hash> = offered.into_iter().filter(|h| !held.contains_key(h)).collect();
815        let Msg::Want { hashes } = swap(dialler, tx, rx, Msg::Want { hashes: want.iter().map(|h| to_hex(h)).collect() }).await? else {
816            return Err("protocol: expected want".into());
817        };
818        let wanted: Vec<Hash> = hashes.iter().map(|h| from_hex::<32>(h)).collect::<Result<Vec<_>, _>>()?.into_iter().filter(|h| offer.contains(h)).collect();
819        let give = signed(&set, &origins, wanted.iter());
820        report.replicated += give.len();
821        self.gate()?; // again just before lines leave
822        let Msg::Lines { lines: got } = swap(dialler, tx, rx, Msg::Lines { lines: give }).await? else {
823            return Err("protocol: expected lines".into());
824        };
825        self.store_lines(&path, got, &want, &mine, Some(remote), &mut report)?;
826
827        // Excerpts by commit: any holder answers, whoever wrote them.
828        let ask: Vec<String> = ask.iter().take(MAX_ASK).cloned().collect();
829        let Msg::Ask { commits: theirs } = swap(dialler, tx, rx, Msg::Ask { commits: ask.clone() }).await? else {
830            return Err("protocol: expected ask".into());
831        };
832        let held = lines::read(&path);
833        let set = origin::admitted(held.clone(), &origin::load(home), &mine);
834        let answer: Vec<&Hash> = set.iter().filter(|(_, l)| about(&theirs, l)).map(|(h, _)| h).take(MAX_ASK).collect();
835        let give = signed(&set, &origin::load(home), answer.into_iter());
836        report.answered += give.len();
837        self.gate()?; // again just before lines leave
838        let Msg::Lines { lines: got } = swap(dialler, tx, rx, Msg::Lines { lines: give }).await? else {
839            return Err("protocol: expected lines".into());
840        };
841        let want: BTreeSet<Hash> = got.iter().filter(|s| about(&ask, &s.line)).map(|s| lines::hash(&s.line)).filter(|h| !held.contains_key(h)).collect();
842        self.store_lines(&path, got, &want, &mine, None, &mut report)?;
843
844        // Live: presence, ephemeral (live.rs).
845        let now = borg::learn::now_ms();
846        let live = Msg::Live { presence: live::outgoing(home, &self.me, now), directives: Directives };
847        self.gate()?; // again just before anything leaves
848        let Msg::Live { presence, .. } = swap(dialler, tx, rx, live).await? else {
849            return Err("protocol: expected live".into());
850        };
851        let _w = self.store.lock().unwrap_or_else(|e| e.into_inner());
852        report.presence += live::receive(home, &mine, &me, presence, now)?;
853        Ok(report)
854    }
855
856    /// One log reconciled with the peer (step 3): the admitted set, recon rounds, then the lines each
857    /// side lacks, signed, stored when their origin verifies and is admitted. `v1`: the peer speaks
858    /// `V13_ALPN` (`offered`).
859    #[allow(clippy::too_many_arguments)] // one step of a session: its stream, its log, what it checks against
860    async fn reconcile(
861        &self,
862        dialler: bool,
863        tx: &mut SendStream,
864        rx: &mut RecvStream,
865        path: &Path,
866        origins: &HashMap<Hash, Origin>,
867        mine: &Membership,
868        v1: bool,
869        report: &mut Report,
870    ) -> Result<(), String> {
871        let set = offered(path, origins, mine, borg::learn::now_ms(), v1);
872        self.gate()?; // read first, then gate: a line written after a switch to local is never sent
873        let mut r = Recon::new(&set);
874        loop {
875            let m = r.message();
876            let Msg::Round { round } = swap(dialler, tx, rx, Msg::Round { round: m.clone() }).await? else {
877                return Err("protocol: expected a reconciliation round".into());
878            };
879            if r.absorb(&m, round)? {
880                break;
881            }
882        }
883        (report.rounds, report.fps, report.hashes) = (report.rounds + r.cost.rounds, report.fps + r.cost.fps, report.hashes + r.cost.hashes);
884        let give = frame(&set, origins, r.give.iter());
885        report.sent += give.len();
886        self.gate()?; // again just before lines leave
887        let Msg::Lines { lines: got } = swap(dialler, tx, rx, Msg::Lines { lines: give }).await? else {
888            return Err("protocol: expected lines".into());
889        };
890        self.store_lines(path, got, &r.want, mine, None, report)
891    }
892
893    /// One log a peer reconciles and this node no longer keeps (the `PLAYBOOKS` step, or a log of a
894    /// 0.3.0 peer's hive goal, 3b), followed in step with it and kept nowhere. `log` names it: the peer,
895    /// then the goal or the file (no goal id holds a dot), then the goal's log. The listener
896    /// answers the peer's first round with that round itself, which closes it with nothing to give
897    /// either way. The dialler speaks first, so it reconciles the hashes this peer gave before (`legacy`)
898    /// and remembers the ones it gives now: the peer sends each line once per process, not per session.
899    async fn skip(&self, dialler: bool, tx: &mut SendStream, rx: &mut RecvStream, log: (String, String, usize)) -> Result<(), String> {
900        let mut want = BTreeSet::new();
901        if dialler {
902            let given = self.legacy.lock().unwrap_or_else(|e| e.into_inner()).get(&log).cloned().unwrap_or_default();
903            let given: BTreeMap<Hash, ()> = given.into_iter().map(|h| (h, ())).collect();
904            let mut r = Recon::new(&given);
905            loop {
906                let m = r.message();
907                let Msg::Round { round } = swap(true, tx, rx, Msg::Round { round: m.clone() }).await? else {
908                    return Err("protocol: expected a reconciliation round".into());
909                };
910                if r.absorb(&m, round)? {
911                    break;
912                }
913            }
914            want = r.want;
915        } else {
916            let Msg::Round { round } = recv(rx).await? else {
917                return Err("protocol: expected a reconciliation round".into());
918            };
919            send(tx, &Msg::Round { round }).await?;
920        }
921        let Msg::Lines { lines: got } = swap(dialler, tx, rx, Msg::Lines { lines: Vec::new() }).await? else {
922            return Err("protocol: expected lines".into());
923        };
924        let got: Vec<Hash> = got.iter().map(|s| lines::hash(&s.line)).filter(|h| want.contains(h)).collect();
925        if !got.is_empty() {
926            self.legacy.lock().unwrap_or_else(|e| e.into_inner()).entry(log).or_default().extend(got);
927        }
928        Ok(())
929    }
930
931    /// Stores what a peer gave (`Store::store_lines`).
932    fn store_lines(
933        &self,
934        path: &Path,
935        got: Vec<Signed>,
936        want: &BTreeSet<Hash>,
937        m: &Membership,
938        author: Option<&str>,
939        report: &mut Report,
940    ) -> Result<(), String> {
941        self.local().store_lines(path, got, want, m, author, report)
942    }
943
944    /// Writes what `stored` decided (`Store::store_messages`).
945    #[cfg(test)]
946    fn store_messages(
947        &self,
948        path: &Path,
949        keep: BTreeMap<Hash, Signed>,
950        decided: (&str, Stored),
951        m: &Membership,
952        now: i64,
953        report: &mut Report,
954    ) -> Result<steer::Admitted, String> {
955        self.local().store_messages(path, keep, decided, m, now, report)
956    }
957}
958
959/// One node's store as a session reads and writes it: where it is, the lock its writes take, and the
960/// key that stamps its lines. `Node` stamps and merges through it alone; built over a store directly,
961/// it does the same without an endpoint, for simulation and tests (tests/two_nodes_cli.rs), so they
962/// run the merge a session runs.
963pub struct Store<'a> {
964    /// The collective's store.
965    pub home: &'a Path,
966    /// Serialises the writes to the store (`Node`'s, shared by the concurrent sessions of `serve`).
967    pub lock: &'a Mutex<()>,
968    /// This node's key: its own lines are stamped with it, and steering it receives admitted under it.
969    pub me: &'a Identity,
970}
971
972impl Store<'_> {
973    /// Stamps the lines this node's hooks wrote since the last call as its own (origin.rs).
974    pub fn stamp(&self) -> Result<usize, String> {
975        let _w = self.lock.lock().unwrap_or_else(|e| e.into_inner());
976        // Fresh outcomes first, so they are stamped in the same step.
977        borg::outcome::publish(self.home);
978        origin::stamp(self.home, self.me, &logs(self.home), borg::learn::now_ms())
979    }
980
981    /// Stores the lines that were asked for, whose origin verifies and is admitted (and, for
982    /// replicas, is the peer itself), and that may travel under it (lines.rs `travels`): origin
983    /// records first, so a line is never re-stamped as ours.
984    pub fn store_lines(
985        &self,
986        path: &Path,
987        got: Vec<Signed>,
988        want: &BTreeSet<Hash>,
989        m: &Membership,
990        author: Option<&str>,
991        report: &mut Report,
992    ) -> Result<(), String> {
993        let (file, now) = (log_name(path), borg::learn::now_ms());
994        let mut keep: BTreeMap<Hash, Signed> = BTreeMap::new();
995        for s in got {
996            let h = lines::hash(&s.line);
997            let ok = want.contains(&h)
998                && s.origin.verify(&s.line).is_ok()
999                && m.admits(&s.origin.key, s.origin.ms)
1000                && author.is_none_or(|a| a == s.origin.key)
1001                && lines::travels(file, &s.line, &s.origin.key, now);
1002            if ok {
1003                keep.insert(h, s);
1004            } else {
1005                report.rejected += 1;
1006            }
1007        }
1008        let _w = self.lock.lock().unwrap_or_else(|e| e.into_inner());
1009        if file != borg::msg::FILE {
1010            let known = origin::load(self.home);
1011            origin::append(self.home, &keep.iter().filter(|(h, _)| !known.contains_key(*h)).map(|(_, s)| s.origin.clone()).collect::<Vec<_>>())?;
1012            report.received += lines::merge(path, &keep.into_values().map(|s| s.line).collect::<Vec<_>>())?;
1013            return Ok(());
1014        }
1015        // Messages: decided over the log as read now (`stored`), before its lock, so a hook's `ack` or a
1016        // `send` never waits on the window's arithmetic.
1017        let text = std::fs::read_to_string(path).unwrap_or_default();
1018        let decided = stored(&text, &keep, now);
1019        let a = self.store_messages(path, keep, (&text, decided), m, now, report)?;
1020        (report.steered, report.dropped) = (report.steered + a.new, report.dropped + a.dropped);
1021        drop(_w);
1022        // Outside both locks: it rereads the log and the outcomes, and a `send` must not wait on it.
1023        if a.new > 0 {
1024            borg::veto::apply_messages(self.home, now);
1025        }
1026        Ok(())
1027    }
1028
1029    /// Writes what `stored` decided over the message log at `path` as it read (`decided`: that text, and
1030    /// what it takes of the offered `keep`), under the log's lock (borg msg.rs `send` and `ack` take it
1031    /// too). A log that a `send` or an `ack` changed since is decided again first, so a line written in
1032    /// between counts in the window and is never lost to a rewrite. A decision that writes nothing takes
1033    /// no lock: it holds as of its read. The write is recorded for the log's readers (msg.rs `wrote`): seen
1034    /// lines alone leave its talk, so they make no hook or waiter read it again. Then the privileged lines
1035    /// taken are admitted (steer.rs `admit`).
1036    fn store_messages(
1037        &self,
1038        path: &Path,
1039        mut keep: BTreeMap<Hash, Signed>,
1040        decided: (&str, Stored),
1041        m: &Membership,
1042        now: i64,
1043        report: &mut Report,
1044    ) -> Result<steer::Admitted, String> {
1045        let (read, first) = decided;
1046        if first.taken.is_empty() && first.body.is_none() {
1047            report.rejected += first.refused;
1048            return Ok(steer::Admitted::default());
1049        }
1050        let known = origin::load(self.home);
1051        borg::live::locked(path, || {
1052            let text = std::fs::read_to_string(path).unwrap_or_default();
1053            let d = if text == read { first } else { stored(&text, &keep, now) };
1054            keep.retain(|h, _| d.taken.contains(h));
1055            report.rejected += d.refused;
1056            let new: Vec<borg::msg::Message> = keep.values().filter_map(|s| serde_json::from_str(&s.line).ok()).collect();
1057            origin::append(self.home, &keep.iter().filter(|(h, _)| !known.contains_key(*h)).map(|(_, s)| s.origin.clone()).collect::<Vec<_>>())?;
1058            let io = |e: std::io::Error| format!("{}: {e}", path.display());
1059            let before = borg::msg::mark_of(path);
1060            let talk = keep.values().any(|s| !matches!(Line::parse(&s.line), Some(Line::Seen(_))));
1061            match &d.body {
1062                Some(body) => {
1063                    borg::steer::rewrite(path, body).map_err(io)?;
1064                    borg::msg::wrote(path, before.as_deref(), borg::msg::Change::rewrite(&text, body, now), talk);
1065                }
1066                None if keep.is_empty() => {}
1067                None => {
1068                    // New to the text read under the lock, so the lines are appended as they are, each on a line of its own.
1069                    let open = !text.is_empty() && !text.ends_with('\n');
1070                    let out: String = std::iter::once(if open { "\n" } else { "" }).chain(keep.values().flat_map(|s| [s.line.as_str(), "\n"])).collect();
1071                    std::fs::OpenOptions::new()
1072                        .create(true)
1073                        .append(true)
1074                        .open(path)
1075                        .and_then(|mut f| std::io::Write::write_all(&mut f, out.as_bytes()))
1076                        .map_err(io)?;
1077                    borg::msg::wrote(path, before.as_deref(), borg::msg::Change::Appended(out.len()), talk);
1078                }
1079            }
1080            report.received += keep.len();
1081            // After the write: a hook that sees the line before its admission shows it at its next call.
1082            steer::admit(self.home, m, &self.me.public(), &new, now).map_err(|e| format!("admissions: {e}"))
1083        })
1084    }
1085}
1086
1087/// What the message log takes of the lines a peer offered (`stored`).
1088struct Stored {
1089    /// The offered lines taken.
1090    taken: BTreeSet<Hash>,
1091    /// How many offered lines the log lacked and refused: outside their window, or cut by the bound.
1092    refused: usize,
1093    /// The log's new body when the bound cuts a held line; otherwise the taken lines are appended.
1094    body: Option<String>,
1095}
1096
1097/// What the message log holding `text` takes at `now` of `keep`, the lines a peer offered that verified
1098/// (`Store::store_lines`): of those it lacks, the ones within the window over held and new together
1099/// (`window`), and of those what the log's bound keeps (borg msg.rs `bound`). A new line the bound cuts
1100/// is refused, so it is not taken again when offered again; a held line it cuts goes. Pure: nodes that
1101/// hold the same lines and are offered the same take the same.
1102fn stored(text: &str, keep: &BTreeMap<Hash, Signed>, now: i64) -> Stored {
1103    let file: Vec<(Hash, &str)> = text.lines().map(|l| (lines::hash(l), l)).collect();
1104    let held: HashSet<&Hash> = file.iter().map(|(h, _)| h).collect();
1105    let new: Vec<(&Hash, &Signed)> = keep.iter().filter(|(h, _)| !held.contains(h)).collect();
1106    let lacked = new.len();
1107    let paced: Vec<(&Hash, &Signed)> = if new.is_empty() {
1108        new
1109    } else {
1110        let all: BTreeMap<Hash, String> = file.iter().map(|(h, l)| (*h, l.to_string())).chain(new.iter().map(|(h, s)| (**h, s.line.clone()))).collect();
1111        let w = window(&all);
1112        new.into_iter().filter(|(h, _)| w.contains(*h)).collect()
1113    };
1114    let (kept, _) = borg::msg::bound(file.iter().map(|(_, l)| *l).chain(paced.iter().map(|(_, s)| s.line.as_str())), now);
1115    let set: HashSet<&str> = kept.iter().copied().collect();
1116    let taken: BTreeSet<Hash> = paced.iter().filter(|(_, s)| set.contains(s.line.as_str())).map(|(h, _)| **h).collect();
1117    let body = file.iter().any(|(_, l)| !set.contains(l)).then(|| kept.iter().map(|l| format!("{l}\n")).collect());
1118    Stored { refused: lacked - taken.len(), taken, body }
1119}
1120
1121/// The lines of the log at `path` this node offers at `now`: `travelling`, and for the messages those
1122/// within their `window`; to a `V13_ALPN` peer (`v1`) only what 0.5.0's window takes of them
1123/// (`v1_window`). Public for simulation and tests: it is the set a session reconciles (step 3).
1124pub fn offered(path: &Path, origins: &HashMap<Hash, Origin>, m: &Membership, now: i64, v1: bool) -> BTreeMap<Hash, String> {
1125    let mut set = travelling(path, origins, m, now);
1126    if log_name(path) == borg::msg::FILE {
1127        let w = window(&lines::read(path));
1128        set.retain(|h, _| w.contains(h));
1129        if v1 {
1130            let w = v1_window(&set, v1_share(m));
1131            set.retain(|h, _| w.contains(h));
1132        }
1133    }
1134    set
1135}
1136
1137/// The lines of the log at `path` that are admitted and may travel under their origin at `now`
1138/// (lines.rs `travels`).
1139fn travelling(path: &Path, origins: &HashMap<Hash, Origin>, m: &Membership, now: i64) -> BTreeMap<Hash, String> {
1140    let file = log_name(path);
1141    let mut set = origin::admitted(lines::read(path), origins, m);
1142    set.retain(|h, l| origins.get(h).is_some_and(|o| lines::travels(file, l, &o.key, now)));
1143    set
1144}
1145
1146/// The message lines of `set` within their pacing: every steer and seen line, and each tell or ask
1147/// within what every window admits (borg msg.rs `within_windows`: acknowledged, or at most
1148/// `WINDOW_CAP` unacknowledged from its sender to its recipient within `V1_RATE_MS` from it). It never
1149/// refuses what a sender's `send` wrote, and it reads no line older than the one it decides, so the TTL
1150/// and the bound never take a held line out of it: what a node holds it offers while the line travels,
1151/// and a node that missed a line takes it later, its acknowledged predecessors pruned or not. It depends
1152/// on the lines alone, so nodes holding the same lines take the same, whatever order they arrived in. A
1153/// line that is not well formed is outside.
1154fn window(set: &BTreeMap<Hash, String>) -> BTreeSet<Hash> {
1155    let (hashes, all): (Vec<Hash>, Vec<Line>) = set.iter().filter_map(|(h, l)| Line::parse(l).map(|x| (*h, x))).unzip();
1156    let admitted = borg::msg::within_windows(&all);
1157    hashes
1158        .into_iter()
1159        .zip(&all)
1160        .filter(|(_, l)| match l {
1161            Line::Seen(_) => true,
1162            Line::Message(m) => m.kind.privileged() || admitted.contains(m.id.as_str()),
1163        })
1164        .map(|(h, _)| h)
1165        .collect()
1166}
1167
1168/// How many messages of one node a `V13_ALPN` peer keeps: an equal share of `V1_KEEP` per member (its
1169/// `per_node`).
1170fn v1_share(m: &Membership) -> usize {
1171    (V1_KEEP / m.members.len().max(1)).max(1)
1172}
1173
1174/// What a `V13_ALPN` peer's window (0.5.0's `window`) takes of `set`: only v1 messages (no v2 text, no
1175/// seen line), per node, sender and recipient one tell or ask per `V1_RATE_MS` (the earliest wins; a
1176/// steer is not paced), then for conversation and for steers apart the `per_node` newest per node and
1177/// the `V1_KEEP` newest of those. Ordered by (ms, hash), as that peer orders them.
1178fn v1_window(set: &BTreeMap<Hash, String>, per_node: usize) -> BTreeSet<Hash> {
1179    let mut all: Vec<(i64, Hash, borg::msg::Message)> = set
1180        .iter()
1181        .filter_map(|(h, l)| serde_json::from_str::<borg::msg::Message>(l).ok().filter(|m| m.v == 1 && m.well_formed()).map(|m| (m.ms, *h, m)))
1182        .collect();
1183    all.sort_by_key(|x| (x.0, x.1));
1184    let mut last: HashMap<(String, String, String), i64> = HashMap::new();
1185    let mut paced: Vec<(Hash, String, bool)> = Vec::new();
1186    for (ms, h, m) in all {
1187        if m.kind.privileged() {
1188            paced.push((h, m.node, true));
1189            continue;
1190        }
1191        let slot = last.entry((m.node.clone(), m.by, m.to)).or_insert(i64::MIN);
1192        if ms.saturating_sub(*slot) >= V1_RATE_MS {
1193            *slot = ms;
1194            paced.push((h, m.node, false));
1195        }
1196    }
1197    let mut per: HashMap<(String, bool), usize> = HashMap::new();
1198    let mut total = [0usize; 2];
1199    paced
1200        .into_iter()
1201        .rev()
1202        .filter(|(_, node, steers)| {
1203            let n = per.entry((node.clone(), *steers)).or_default();
1204            *n += 1;
1205            *n <= per_node
1206        })
1207        .filter(|(_, _, steers)| {
1208            let t = &mut total[usize::from(*steers)];
1209            *t += 1;
1210            *t <= V1_KEEP
1211        })
1212        .map(|(h, _, _)| h)
1213        .collect()
1214}
1215
1216/// A log's file name, as lines.rs `SYNCED` lists it.
1217fn log_name(path: &Path) -> &str {
1218    path.file_name().and_then(|f| f.to_str()).unwrap_or_default()
1219}
1220
1221/// The goals a 0.3.0 peer reconciles after naming `ids` while this node names none, in its order: the
1222/// distinct valid goal ids (its `goal::valid_id`: `[a-z0-9-]{1,40}`), sorted.
1223fn legacy_goals(ids: Vec<String>) -> BTreeSet<String> {
1224    let valid = |id: &String| (1..=40).contains(&id.len()) && id.bytes().all(|b| b.is_ascii_lowercase() || b.is_ascii_digit() || b == b'-');
1225    ids.into_iter().filter(valid).collect()
1226}
1227
1228/// Where a dialler that announced `listen` can be reached: the IP its packets came from (what
1229/// QUIC observed) with the port it listens on. Nothing for an address nobody could dial.
1230fn reachable(listen: &str, observed: Option<SocketAddr>) -> Option<SocketAddr> {
1231    let listen: SocketAddr = listen.parse().ok()?;
1232    let ip = match observed {
1233        Some(o) => o.ip().to_canonical(),
1234        None => listen.ip(),
1235    };
1236    let at = SocketAddr::new(ip, listen.port());
1237    (!at.ip().is_unspecified() && at.port() != 0).then_some(at)
1238}
1239
1240/// Remembers a member that dialled in where it can be dialled back: the IP its packets came from
1241/// with the port it listens on, and the home relay it announced. Known by key at least.
1242fn heard(home: &Path, key: &str, listen: Option<String>, relay: Option<String>, observed: Option<SocketAddr>) {
1243    let direct = listen.and_then(|l| reachable(&l, observed)).map(|a| a.to_string());
1244    let hints: Vec<String> = direct.into_iter().chain(relay.filter(|r| addr::hints(r).is_ok())).collect();
1245    let _ = peers::remember(home, key, &hints.join(","));
1246}
1247
1248/// Whether a line of reasoning.jsonl is about one of `commits`: an excerpt by its `commit`, a
1249/// decision → fix link record (borg notes/links.rs) by either side, so fetching the fix brings it too.
1250fn about(commits: &[String], line: &str) -> bool {
1251    let Ok(v) = serde_json::from_str::<serde_json::Value>(line) else { return false };
1252    ["commit", "fix"].iter().filter_map(|k| v.get(k)?.as_str()).any(|c| asked(commits, c))
1253}
1254
1255/// Whether `commit` is one of `asked` (full ids, or abbreviations of at least 7 characters).
1256fn asked(asked: &[String], commit: &str) -> bool {
1257    asked.iter().any(|a| a == commit || (a.len() >= 7 && commit.starts_with(a.as_str())))
1258}
1259
1260/// The `Lines` frame a session gives of a log after its reconciliation (step 3): the lines of `set`
1261/// (`offered`) named by `hashes`, the ones the peer lacks (`Recon::give`), each with its signed origin,
1262/// as many as the frame's budget takes (`within_budget`). Public for simulation and tests, which hand
1263/// it to the peer instead of a stream.
1264pub fn frame<'a>(set: &BTreeMap<Hash, String>, origins: &HashMap<Hash, Origin>, hashes: impl Iterator<Item = &'a Hash>) -> Vec<Signed> {
1265    within_budget(signed(set, origins, hashes))
1266}
1267
1268/// The first of `lines` whose sizes add up to at most `MAX_GIVE` (at least one).
1269fn within_budget(lines: Vec<Signed>) -> Vec<Signed> {
1270    let mut used = 0;
1271    lines
1272        .into_iter()
1273        .enumerate()
1274        .take_while(|(i, s)| {
1275            used += s.line.len();
1276            *i == 0 || used <= MAX_GIVE
1277        })
1278        .map(|(_, s)| s)
1279        .collect()
1280}
1281
1282fn signed<'a>(set: &BTreeMap<Hash, String>, origins: &HashMap<Hash, Origin>, hashes: impl Iterator<Item = &'a Hash>) -> Vec<Signed> {
1283    hashes.filter_map(|h| Some(Signed { line: set.get(h)?.clone(), origin: origins.get(h)?.clone() })).collect()
1284}
1285
1286/// The members (other than `author`) that hold a replica of excerpt `h`: the `REPLICAS` highest
1287/// `blake3(h, key)` (rendezvous hashing: every node computes the same choice, and a membership
1288/// change moves only the excerpts of the members it touches).
1289pub fn replicas(h: &Hash, m: &Membership, author: &str) -> Vec<String> {
1290    let mut keys: Vec<(Hash, &str)> = m
1291        .members
1292        .iter()
1293        .filter(|x| x.key != author)
1294        .map(|x| {
1295            let mut s = blake3::Hasher::new();
1296            s.update(h);
1297            s.update(x.key.as_bytes());
1298            (*s.finalize().as_bytes(), x.key.as_str())
1299        })
1300        .collect();
1301    keys.sort_by(|a, b| b.cmp(a));
1302    keys.into_iter().take(REPLICAS).map(|(_, k)| k.to_string()).collect()
1303}
1304
1305fn members(home: &Path) -> Result<Membership, String> {
1306    Membership::load(home)?
1307        .ok_or_else(|| format!("no membership in {}: `borg mesh init` (admin) or `borg mesh join <file>`", crate::membership::path(home).display()))
1308}
1309
1310/// The key iroh authenticated, if the membership lists it.
1311fn authorise(conn: &Connection, membership: &Membership, name: &str) -> Result<String, String> {
1312    let remote = to_hex(conn.remote_id().as_bytes());
1313    match membership.role_of(&remote) {
1314        Some(_) => Ok(remote),
1315        None => Err(format!("rejected {remote}: not a member of {name}")),
1316    }
1317}
1318
1319/// The dialler sends then reads; the listener reads (and could refuse) then sends.
1320async fn swap(dialler: bool, tx: &mut SendStream, rx: &mut RecvStream, m: Msg) -> Result<Msg, String> {
1321    if dialler {
1322        send(tx, &m).await?;
1323        recv(rx).await
1324    } else {
1325        let theirs = recv(rx).await?;
1326        send(tx, &m).await?;
1327        Ok(theirs)
1328    }
1329}
1330
1331async fn send(tx: &mut SendStream, m: &Msg) -> Result<(), String> {
1332    let body = serde_json::to_vec(m).map_err(|e| e.to_string())?;
1333    tx.write_all(&(body.len() as u32).to_be_bytes()).await.map_err(|e| format!("send: {e}"))?;
1334    tx.write_all(&body).await.map_err(|e| format!("send: {e}"))
1335}
1336
1337async fn recv(rx: &mut RecvStream) -> Result<Msg, String> {
1338    let mut len = [0u8; 4];
1339    rx.read_exact(&mut len).await.map_err(|e| format!("receive: {e}"))?;
1340    let len = u32::from_be_bytes(len) as usize;
1341    if len > MAX_FRAME {
1342        return Err(format!("frame of {len} bytes exceeds {MAX_FRAME}"));
1343    }
1344    let mut body = vec![0u8; len];
1345    rx.read_exact(&mut body).await.map_err(|e| format!("receive: {e}"))?;
1346    match serde_json::from_slice(&body).map_err(|e| format!("protocol: {e}"))? {
1347        Msg::Refused { reason } => Err(format!("refused by peer: {reason}")),
1348        m => Ok(m),
1349    }
1350}
1351
1352#[cfg(test)]
1353mod tests {
1354    use super::*;
1355    use serde_json::json;
1356
1357    /// An excerpt is asked for by its commit; a link record by its decision or by its fix.
1358    #[test]
1359    fn lines_about_the_asked_commits() {
1360        let ask = vec!["d0d0d0d".to_string(), "f1f1f1f1f1".to_string()];
1361        let excerpt = json!({"kind": "reasoning", "commit": "d0d0d0d0d0"}).to_string();
1362        let link = |c: &str, f: &str| json!({"kind": "regress", "commit": c, "fix": f}).to_string();
1363        assert!(about(&ask, &excerpt) && about(&ask, &link("aaaaaaaaaa", "f1f1f1f1f1")) && about(&ask, &link("d0d0d0d0d0", "bbbbbbbbbb")));
1364        assert!(!about(&ask, &link("aaaaaaaaaa", "bbbbbbbbbb")) && !about(&ask, "not json"));
1365    }
1366
1367    /// A frame of lines stays within `MAX_GIVE`: the rest wait for the next session; one line always goes.
1368    #[test]
1369    fn a_frame_gives_lines_up_to_its_budget() {
1370        let me = Identity::load_or_create(&std::env::temp_dir().join(format!("borg-mesh-budget-{}", std::process::id()))).unwrap();
1371        let line = |n: usize| {
1372            let l = "x".repeat(n);
1373            Signed { origin: Origin::new(&me, &lines::hash(&l), 1), line: l }
1374        };
1375        let give = within_budget(vec![line(MAX_GIVE / 2), line(MAX_GIVE / 2), line(1)]);
1376        assert_eq!(give.len(), 2);
1377        assert_eq!(within_budget(vec![line(MAX_GIVE + 1), line(1)]).len(), 1);
1378        assert!(within_budget(Vec::new()).is_empty());
1379    }
1380
1381    /// One node's machine: a root store (`BORG_HOME`), a config (`BORG_CONFIG`) and a repo that
1382    /// routes to collective "work" with `share`.
1383    fn node(tag: &str, share: &str) -> Place {
1384        let base = std::env::temp_dir().join(format!("borg-mesh-sync-{tag}-{}", std::process::id()));
1385        let _ = std::fs::remove_dir_all(&base);
1386        std::fs::create_dir_all(home_of(&base)).unwrap();
1387        let p = Place { config: base.join("collectives.json"), base: base.clone(), dir: base.join("repo").display().to_string(), collective: None };
1388        configure(&p, share);
1389        p
1390    }
1391
1392    fn configure(p: &Place, share: &str) {
1393        let cfg = json!({"default": "off", "collectives": [{"name": "work", "paths": [p.dir], "share": share, "relay": "disabled"}]});
1394        std::fs::write(&p.config, cfg.to_string()).unwrap();
1395    }
1396
1397    fn home_of(base: &Path) -> PathBuf {
1398        base.join("collectives").join("work")
1399    }
1400
1401    fn home(p: &Place) -> PathBuf {
1402        home_of(&p.base)
1403    }
1404
1405    fn lesson(id: &str, version: u32, q: &str) -> String {
1406        json!({"id": id, "origin_ms": 1, "question": q, "version": version}).to_string()
1407    }
1408
1409    fn write(home: &Path, file: &str, lines: &[String]) {
1410        std::fs::write(home.join(file), lines.iter().map(|l| format!("{l}\n")).collect::<String>()).unwrap();
1411    }
1412
1413    const LOCAL: &str = "127.0.0.1:0";
1414
1415    /// Alice founds the collective and invites Bob, who installs the document she hands him.
1416    fn team(tag: &str) -> (Place, Place) {
1417        let (a, b) = (node(&format!("{tag}-alice"), "team"), node(&format!("{tag}-bob"), "team"));
1418        let alice = Identity::load_or_create(&home(&a)).unwrap();
1419        let bob = Identity::load_or_create(&home(&b)).unwrap();
1420        let mut m = Membership::found("work", &alice);
1421        m.invite(&alice, &bob.public(), "member").unwrap();
1422        m.save(&home(&a)).unwrap();
1423        m.save(&home(&b)).unwrap();
1424        (a, b)
1425    }
1426
1427    fn rollup(node: &str, version: u64, turns: u64) -> String {
1428        json!({"id": format!("stat:{node}:dev@x:2026-10-12"), "version": version, "day": "2026-10-12", "node": node,
1429            "engineer": "dev@x", "counts": {"turn_end": turns}, "usd": 0.0, "dur_ms": 0})
1430        .to_string()
1431    }
1432
1433    #[tokio::test(flavor = "multi_thread")]
1434    async fn usage_rollups_sync_and_the_latest_version_wins() {
1435        let (pa, pb) = team("stats");
1436        let (a, b) = (home(&pa), home(&pb));
1437        write(&a, "stats.jsonl", &[rollup("na", 1, 3), rollup("na", 2, 5)]);
1438        write(&b, "stats.jsonl", &[rollup("nb", 1, 7)]);
1439        let alice = Node::bind(&pa, LOCAL.parse().unwrap()).await.unwrap();
1440        let bob = Node::bind(&pb, LOCAL.parse().unwrap()).await.unwrap();
1441        let ticket = alice.ticket();
1442        let (served, synced) = tokio::join!(alice.accept(), bob.sync(&ticket));
1443        let (served, synced) = (served.unwrap(), synced.unwrap());
1444        assert_eq!((synced.received, served.received), (2, 1), "both of alice's lines reach bob, bob's one reaches alice");
1445        for h in [&a, &b] {
1446            let r = borg::stats::rollups(h);
1447            assert_eq!(r.len(), 2, "one bucket per node");
1448            assert_eq!(r["stat:na:dev@x:2026-10-12"].counts["turn_end"], 5, "the newest version of alice's bucket");
1449            assert_eq!(r["stat:nb:dev@x:2026-10-12"].counts["turn_end"], 7);
1450        }
1451    }
1452
1453    /// A store upgraded from 0.4.0 still holds playbooks.jsonl, stamped: it is neither offered nor
1454    /// taken, and the session still steps through it, so the two nodes stay in order.
1455    #[tokio::test(flavor = "multi_thread")]
1456    async fn a_leftover_playbooks_log_is_not_synced() {
1457        let (pa, pb) = team("playbooks");
1458        let (a, b) = (home(&pa), home(&pb));
1459        let edit = json!({"pb": "p1", "edit": "e1", "step": "s1", "text": "run the parser tests", "ms": 1}).to_string();
1460        write(&a, PLAYBOOKS, std::slice::from_ref(&edit));
1461        let h = lines::hash(&edit);
1462        origin::append(&a, &[Origin::new(&Identity::load_or_create(&a).unwrap(), &h, 1)]).unwrap();
1463        let c = json!({"id": "cabc", "text": "locks are taken in id order", "lesson": "L1", "by": "lead@x", "ms": 5, "source": "veto"}).to_string();
1464        write(&a, borg::veto::FILE, std::slice::from_ref(&c));
1465        let alice = Node::bind(&pa, LOCAL.parse().unwrap()).await.unwrap();
1466        let bob = Node::bind(&pb, LOCAL.parse().unwrap()).await.unwrap();
1467        let ticket = alice.ticket();
1468        let (served, synced) = tokio::join!(alice.accept(), bob.sync(&ticket));
1469        let (served, synced) = (served.unwrap(), synced.unwrap());
1470        assert_eq!((synced.received, served.sent), (1, 1), "only the constraint, the step after the playbooks', moves");
1471        assert!(holds(&b, borg::veto::FILE, &c));
1472        assert!(!b.join(PLAYBOOKS).exists(), "no playbook reaches bob");
1473        assert!(holds(&a, PLAYBOOKS, &edit), "alice's leftover stays untouched");
1474        alice.close().await;
1475        bob.close().await;
1476    }
1477
1478    #[tokio::test(flavor = "multi_thread")]
1479    async fn a_ratified_constraint_reaches_the_team_signed() {
1480        let (pa, pb) = team("constraints");
1481        let (a, b) = (home(&pa), home(&pb));
1482        write(&a, "lessons.jsonl", &[lesson("L1", 1, "lock order?")]);
1483        let c = json!({"id": "cabc", "text": "locks are taken in id order", "lesson": "L1", "by": "lead@x", "ms": 5, "source": "veto"}).to_string();
1484        write(&a, borg::veto::FILE, std::slice::from_ref(&c));
1485        let alice = Node::bind(&pa, LOCAL.parse().unwrap()).await.unwrap();
1486        let bob = Node::bind(&pb, LOCAL.parse().unwrap()).await.unwrap();
1487        let ticket = alice.ticket();
1488        let (served, synced) = tokio::join!(alice.accept(), bob.sync(&ticket));
1489        served.unwrap();
1490        assert!(synced.unwrap().received >= 2, "the lesson and the constraint");
1491        assert_eq!(lines::read(&b.join(borg::veto::FILE)).into_values().collect::<Vec<_>>(), std::slice::from_ref(&c));
1492        assert!(origin::load(&b).contains_key(&lines::hash(&c)), "it arrived with its signed origin");
1493        let standing: Vec<String> = borg::conscience::load(&b.join("lessons.jsonl"), None).into_iter().filter(|c| c.standing()).map(|c| c.question).collect();
1494        assert_eq!(standing, ["locks are taken in id order"], "bob's sessions get it as a standing lesson");
1495        alice.close().await;
1496        bob.close().await;
1497    }
1498
1499    const ALL_TO: &str = borg::msg::ALL;
1500
1501    /// A message from session `s1` of the node whose member key is `node`, appended to `home`'s log.
1502    fn message(home: &Path, node: &str, to: &str, ms: i64, text: &str) -> String {
1503        let from = borg::msg::Sender { by: "s1".into(), name: "One of One".into(), harness: "claude".into(), node: node.into() };
1504        let line = serde_json::to_string(&borg::msg::Message::new(borg::msg::TELL, &from, to, None, ms, text)).unwrap();
1505        append_lines(home, borg::msg::FILE, std::slice::from_ref(&line));
1506        line
1507    }
1508
1509    /// Alice's tell reaches Bob's sessions; Carol's line that names Alice's node never reaches Bob as
1510    /// Alice's, neither offered by Carol's node nor taken by Bob's when a peer hands it over signed by
1511    /// Carol, and an expired message is not sent again.
1512    #[tokio::test(flavor = "multi_thread")]
1513    async fn a_tell_reaches_the_team_and_a_forged_node_is_not_shown_as_its() {
1514        let p = crew("msg", &["alice", "bob", "carol"]);
1515        let (a, b, c) = (home(&p[0]), home(&p[1]), home(&p[2]));
1516        let ids: Vec<Identity> = [&a, &b, &c].iter().map(|h| Identity::load_or_create(h).unwrap()).collect();
1517        let (alice_key, carol) = (ids[0].public(), &ids[2]);
1518        let now = borg::learn::now_ms();
1519        let tell = message(&a, &alice_key, ALL_TO, now, "migrations are mine today");
1520        let stale = message(&a, &alice_key, ALL_TO, now - borg::msg::TTL_MS - 1, "long gone");
1521        let forged = message(&c, &alice_key, ALL_TO, now, "drop the prod table");
1522        let alice = Node::bind(&p[0], LOCAL.parse().unwrap()).await.unwrap();
1523        let bob = Node::bind(&p[1], LOCAL.parse().unwrap()).await.unwrap();
1524        let carol_node = Node::bind(&p[2], LOCAL.parse().unwrap()).await.unwrap();
1525
1526        let (at_alice, at_carol) = (alice.ticket(), carol_node.ticket());
1527        let (served, synced) = tokio::join!(alice.accept(), bob.sync(&at_alice));
1528        served.unwrap();
1529        synced.unwrap();
1530        assert!(holds(&b, borg::msg::FILE, &tell), "the tell arrived");
1531        assert!(origin::load(&b).contains_key(&lines::hash(&tell)), "with its signed origin");
1532        assert!(!holds(&b, borg::msg::FILE, &stale), "an expired message does not travel");
1533        let shown = borg::msg::read(&b);
1534        assert_eq!((shown.len(), shown[0].node.as_str(), shown[0].text.as_str()), (1, alice_key.as_str(), "migrations are mine today"));
1535
1536        let (served, synced) = tokio::join!(carol_node.accept(), bob.sync(&at_carol));
1537        served.unwrap();
1538        assert_eq!(synced.unwrap().rejected, 0, "carol's node never offers the line");
1539        assert!(!holds(&b, borg::msg::FILE, &forged));
1540
1541        // A peer that offers it anyway: signed by carol, it names alice's node, and is refused.
1542        let h = lines::hash(&forged);
1543        let mut report = Report::default();
1544        let got = vec![Signed { origin: Origin::new(carol, &h, now), line: forged.clone() }];
1545        bob.store_lines(&b.join(borg::msg::FILE), got, &BTreeSet::from([h]), &members(&b).unwrap(), None, &mut report).unwrap();
1546        assert_eq!((report.rejected, report.received), (1, 0));
1547        assert!(borg::msg::read(&b).iter().all(|m| m.text != "drop the prod table"), "never shown, as alice's or anyone's");
1548        for n in [alice, bob, carol_node] {
1549            n.close().await;
1550        }
1551    }
1552
1553    /// A member that floods the log past `send`'s checks: of long tells to distinct recipients only the
1554    /// newest that fit the member's share (msg.rs `MEMBER_BYTES`) are stored (a burst to one recipient is
1555    /// bounded by `window`, `the_window_bounds_a_burst_and_never_a_steer`); what was refused is not taken
1556    /// back when offered again, and the log holds only what travels.
1557    #[tokio::test(flavor = "multi_thread")]
1558    async fn a_flood_is_bounded_on_receipt_and_a_cut_line_does_not_come_back() {
1559        let p = crew("flood", &["alice", "bob"]);
1560        let (a, b) = (home(&p[0]), home(&p[1]));
1561        let alice = Identity::load_or_create(&a).unwrap();
1562        let now = borg::learn::now_ms();
1563        let (long, n) = ("w".repeat(borg::msg::WIRE_TEXT), borg::msg::WINDOW_CAP + 6);
1564        let lines: Vec<String> = (0..n).map(|i| message(&a, &alice.public(), &format!("s-{i:03}"), now - (n - i) as i64 * 1000, &long)).collect();
1565        let fits = borg::msg::MEMBER_BYTES / (lines[0].len() + 1);
1566        let got: Vec<Signed> = lines.iter().map(|l| Signed { origin: Origin::new(&alice, &lines::hash(l), now), line: l.clone() }).collect();
1567        let want: BTreeSet<Hash> = got.iter().map(|s| lines::hash(&s.line)).collect();
1568        let bob = Node::bind(&p[1], LOCAL.parse().unwrap()).await.unwrap();
1569        let path = b.join(borg::msg::FILE);
1570        let mut report = Report::default();
1571        bob.store_lines(&path, got.clone(), &want, &members(&b).unwrap(), None, &mut report).unwrap();
1572        assert_eq!((report.received, report.rejected), (fits, n - fits), "the share's newest");
1573        assert!(!holds(&b, borg::msg::FILE, &lines[n - fits - 1]) && holds(&b, borg::msg::FILE, &lines[n - fits]), "the oldest past the share");
1574
1575        let mut twice = Report::default();
1576        bob.store_lines(&path, got, &want, &members(&b).unwrap(), None, &mut twice).unwrap();
1577        assert_eq!((twice.received, twice.rejected), (0, n - fits), "nothing cut comes back");
1578        assert_eq!(lines::read(&path).len(), fits);
1579        assert_eq!(offered(&path, &origin::load(&b), &members(&b).unwrap(), now, false).len(), fits);
1580        bob.close().await;
1581    }
1582
1583    /// A node's lines for its own sessions (`local`, which never travel) have their own share of its
1584    /// log (borg msg.rs `bound`): however many it holds, it keeps the travelling lines its peer keeps,
1585    /// so no session offers back a line it cut and has it refused again, session after session.
1586    #[tokio::test(flavor = "multi_thread")]
1587    async fn a_nodes_own_lines_never_cut_what_its_peers_keep() {
1588        let (pa, pb) = team("local-share");
1589        let (a, b) = (home(&pa), home(&pb));
1590        let alice = Identity::load_or_create(&a).unwrap().public();
1591        let bob = Identity::load_or_create(&b).unwrap().public();
1592        let now = borg::learn::now_ms();
1593        let long = "w".repeat(borg::msg::WIRE_TEXT);
1594        let n = borg::msg::WINDOW_CAP / 2 + borg::msg::WINDOW_CAP / 8;
1595        let lines_to = |to: &str, at: i64| -> Vec<String> { (0..n).map(|i| line_of(&alice, &format!("s-{i:03}"), to, at + i as i64, &long).1).collect() };
1596        let out = lines_to(&format!("r@{bob}"), now - 20_000);
1597        let here = lines_to(&format!("local@{alice}"), now - 10_000);
1598        let size = out[0].len().max(here[0].len()) + 1;
1599        assert!(n * size <= borg::msg::MEMBER_BYTES && 2 * n * size > borg::msg::MEMBER_BYTES, "each kind alone fits a share, both do not");
1600        append_lines(&a, borg::msg::FILE, &out);
1601        let (alice_node, bob_node) = (Node::bind(&pa, LOCAL.parse().unwrap()).await.unwrap(), Node::bind(&pb, LOCAL.parse().unwrap()).await.unwrap());
1602        let ticket = alice_node.ticket();
1603        let (served, synced) = tokio::join!(alice_node.accept(), bob_node.sync(&ticket));
1604        assert_eq!((served.unwrap().rejected, synced.unwrap().rejected), (0, 0));
1605        assert!(out.iter().all(|l| holds(&b, borg::msg::FILE, l)), "bob keeps every tell");
1606        // Alice's sessions talk among themselves; an acknowledgement bounds her log as `send` does.
1607        append_lines(&a, borg::msg::FILE, &here);
1608        let first: borg::msg::Message = serde_json::from_str(&here[0]).unwrap();
1609        borg::msg::ack(&a, "s-001", &[&first.id], now).unwrap();
1610        assert!(out.iter().chain(&here).all(|l| holds(&a, borg::msg::FILE, l)), "alice keeps what travels and what stays");
1611        for _ in 0..2 {
1612            let (served, synced) = tokio::join!(alice_node.accept(), bob_node.sync(&ticket));
1613            assert_eq!((served.unwrap().rejected, synced.unwrap().rejected), (0, 0), "nothing offered back, nothing refused");
1614        }
1615        alice_node.close().await;
1616        bob_node.close().await;
1617    }
1618
1619    /// A send or an acknowledgement that writes the log after a session decided what it takes and
1620    /// before it holds the lock: the decision is made again over the log as it is then, so that line
1621    /// is kept by the rewrite (here the bound prunes an expired line) and counts in the windows.
1622    #[tokio::test(flavor = "multi_thread")]
1623    async fn a_line_written_between_the_decision_and_the_lock_is_kept() {
1624        let (pa, pb) = team("race");
1625        let (a, b) = (home(&pa), home(&pb));
1626        let alice = Identity::load_or_create(&a).unwrap();
1627        let bob = Identity::load_or_create(&b).unwrap().public();
1628        let now = borg::learn::now_ms();
1629        let expired = line_of(&alice.public(), "s1", &format!("r0@{bob}"), now - borg::msg::TTL_MS - borg::steer::SLACK_MS - 1, "gone").1;
1630        // As many unread tells to r1 as any window lets through, and an earlier one a peer offers.
1631        let burst: Vec<(borg::msg::Message, String)> =
1632            (0..borg::msg::WINDOW_CAP).map(|i| line_of(&alice.public(), "s1", &format!("r1@{bob}"), now - 2000 + i as i64, &format!("t{i}"))).collect();
1633        let next = line_of(&alice.public(), "s1", &format!("r1@{bob}"), now - 3000, "earlier").1;
1634        let other = line_of(&alice.public(), "s2", &format!("r2@{bob}"), now - 1000, "other").1;
1635        append_lines(&b, borg::msg::FILE, &std::iter::once(expired.clone()).chain(burst.iter().map(|(_, l)| l.clone())).collect::<Vec<_>>());
1636        let path = b.join(borg::msg::FILE);
1637        let keep: BTreeMap<Hash, Signed> =
1638            [&next, &other].into_iter().map(|l| (lines::hash(l), Signed { origin: Origin::new(&alice, &lines::hash(l), now), line: l.clone() })).collect();
1639        let read = std::fs::read_to_string(&path).unwrap();
1640        let decided = stored(&read, &keep, now);
1641        assert_eq!((decided.taken.len(), decided.refused, decided.body.is_some()), (1, 1, true), "one more unread tell to r1 than any window lets through");
1642        // Bob's r1 is shown one of them in between.
1643        let seen = seen_of(&bob, "r1", now - 500, &[&burst[0].0.id]);
1644        append_lines(&b, borg::msg::FILE, std::slice::from_ref(&seen));
1645        let bob_node = Node::bind(&pb, LOCAL.parse().unwrap()).await.unwrap();
1646        let mut report = Report::default();
1647        bob_node.store_messages(&path, keep, (&read, decided), &members(&b).unwrap(), now, &mut report).unwrap();
1648        assert!(holds(&b, borg::msg::FILE, &seen), "the acknowledgement written in between stays");
1649        assert!(holds(&b, borg::msg::FILE, &next) && holds(&b, borg::msg::FILE, &other), "and it let the earlier tell in");
1650        assert!(!holds(&b, borg::msg::FILE, &expired), "the bound still pruned");
1651        assert_eq!((report.received, report.rejected), (2, 0));
1652        bob_node.close().await;
1653    }
1654
1655    /// A message of `node`'s session `by` to `to` at `ms`, as a line.
1656    fn line_of(node: &str, by: &str, to: &str, ms: i64, text: &str) -> (borg::msg::Message, String) {
1657        let from = borg::msg::Sender { by: by.into(), name: "One of One".into(), harness: "claude".into(), node: node.into() };
1658        let m = borg::msg::Message::new(borg::msg::TELL, &from, to, None, ms, text);
1659        let line = serde_json::to_string(&m).unwrap();
1660        (m, line)
1661    }
1662
1663    /// A seen line of `node`'s session `by` for `ids`.
1664    fn seen_of(node: &str, by: &str, ms: i64, ids: &[&str]) -> String {
1665        serde_json::to_string(&borg::msg::Seen::new(by, node, ms, ids)).unwrap()
1666    }
1667
1668    /// A recipient's seen line travels to the sender's node and opens its window there; a seen line
1669    /// another node writes for that session does not.
1670    #[tokio::test(flavor = "multi_thread")]
1671    async fn a_seen_line_from_the_recipients_node_opens_the_senders_window() {
1672        let p = crew("seen", &["alice", "bob", "carol"]);
1673        let (a, b, c) = (home(&p[0]), home(&p[1]), home(&p[2]));
1674        let keys: Vec<String> = [&a, &b, &c].iter().map(|h| Identity::load_or_create(h).unwrap().public()).collect();
1675        let now = borg::learn::now_ms();
1676        let to = format!("s2@{}", keys[1]);
1677        let from = borg::msg::Sender { by: "s1".into(), name: "One of One".into(), harness: "claude".into(), node: keys[0].clone() };
1678        let tell = |ms: i64, text: &str| borg::msg::send(&a, borg::msg::Message::new(borg::msg::TELL, &from, &to, None, ms, text), ms);
1679        let x = tell(now, "one").unwrap();
1680        assert!(matches!(tell(now + 1, "two"), Err(borg::msg::SendError::Window { outstanding: 1, .. })));
1681        let (alice, bob, carol) = (
1682            Node::bind(&p[0], LOCAL.parse().unwrap()).await.unwrap(),
1683            Node::bind(&p[1], LOCAL.parse().unwrap()).await.unwrap(),
1684            Node::bind(&p[2], LOCAL.parse().unwrap()).await.unwrap(),
1685        );
1686        let ticket = alice.ticket();
1687        // Carol's node claims s2 saw it: its own line, which travels, and acknowledges nothing.
1688        let forged = seen_of(&keys[2], "s2", now, &[&x.id]);
1689        append_lines(&c, borg::msg::FILE, std::slice::from_ref(&forged));
1690        let (served, synced) = tokio::join!(alice.accept(), carol.sync(&ticket));
1691        served.unwrap();
1692        synced.unwrap();
1693        assert!(holds(&a, borg::msg::FILE, &forged), "a seen line travels under its node");
1694        assert!(matches!(tell(now + 2, "two"), Err(borg::msg::SendError::Window { .. })), "but s2 is bob's");
1695        // Bob's s2 is shown the tell: its seen line reaches alice, and the window opens.
1696        let (served, synced) = tokio::join!(alice.accept(), bob.sync(&ticket));
1697        served.unwrap();
1698        synced.unwrap();
1699        let caps = borg::harness::by_name("claude").unwrap().caps();
1700        let out = borg::msg::deliver(&b, caps, "prompt", &json!({"session_id": "s2"}), Some(now - 10), now + 3, None).expect("shown");
1701        assert_eq!(out["shown_ids"], json!([x.id]));
1702        let (served, synced) = tokio::join!(alice.accept(), bob.sync(&ticket));
1703        served.unwrap();
1704        assert_eq!(synced.unwrap().rejected, 0);
1705        tell(now + 4, "two").expect("acknowledged on bob's node");
1706        assert_eq!(borg::msg::read(&b), vec![x], "no seen line is read as a message");
1707        for n in [alice, bob, carol] {
1708            n.close().await;
1709        }
1710    }
1711
1712    /// A tell admitted when written still travels once the acknowledged message that opened its window is
1713    /// pruned: alice's s1 had its first tell to s2 acknowledged, so its window let two more through over an
1714    /// hour later; past the first one's TTL alice's log holds only those two, over which `window_for`
1715    /// would admit one, and bob, who missed them, takes both from alice, and carol both from bob.
1716    #[tokio::test(flavor = "multi_thread")]
1717    async fn a_tell_admitted_when_written_travels_after_what_opened_its_window_is_pruned() {
1718        use borg::msg::{Author, Message, SendError, Sender, TELL, TTL_MS};
1719        let p = crew("pruned", &["alice", "bob", "carol"]);
1720        let (a, b, c) = (home(&p[0]), home(&p[1]), home(&p[2]));
1721        let key = Identity::load_or_create(&a).unwrap().public();
1722        let now = borg::learn::now_ms();
1723        let s1 = Sender { by: "s1".into(), name: "One of One".into(), harness: "claude".into(), node: key.clone() };
1724        let to = format!("s2@{key}");
1725        let tell = |ms: i64, text: &str| borg::msg::send(&a, Message::new(TELL, &s1, &to, None, ms, text), ms);
1726        // Past its TTL and the bound's slack by now; the two after it an hour and more later, still fresh.
1727        let t = now - TTL_MS - borg::steer::SLACK_MS - 60_000;
1728        let first = tell(t, "first").unwrap();
1729        borg::msg::ack(&a, "s2", &[&first.id], t + 1).unwrap();
1730        let t2 = t + 2 * borg::steer::SLACK_MS;
1731        let (second, third) = (tell(t2, "second").unwrap(), tell(t2 + 1, "third").unwrap());
1732        assert!(matches!(tell(t2 + 2, "fourth"), Err(SendError::Window { outstanding: 2, window: 2, .. })), "the acknowledgement doubled it");
1733        // Any write now prunes what is past its TTL: here another session's acknowledgement.
1734        borg::msg::ack(&a, "s7", &[&second.id], now).unwrap();
1735        let held = borg::msg::lines(&a);
1736        assert!(held.iter().all(|l| l.message().is_none_or(|m| m.id != first.id)), "the first is pruned");
1737        let w = borg::msg::window_for(&held, Author { node: &key, by: "s1" }, &to, now);
1738        assert!(w.admits(&second.id) && !w.admits(&third.id), "a replay over what is left admits less than send did");
1739        let (alice, bob, carol) = (
1740            Node::bind(&p[0], LOCAL.parse().unwrap()).await.unwrap(),
1741            Node::bind(&p[1], LOCAL.parse().unwrap()).await.unwrap(),
1742            Node::bind(&p[2], LOCAL.parse().unwrap()).await.unwrap(),
1743        );
1744        let line = |m: &Message| serde_json::to_string(m).unwrap();
1745        let (from_alice, from_bob) = (alice.ticket(), bob.ticket());
1746        let (served, synced) = tokio::join!(alice.accept(), bob.sync(&from_alice));
1747        served.unwrap();
1748        assert_eq!(synced.unwrap().rejected, 0);
1749        assert!(holds(&b, borg::msg::FILE, &line(&second)) && holds(&b, borg::msg::FILE, &line(&third)), "bob takes both from alice");
1750        let (served, synced) = tokio::join!(bob.accept(), carol.sync(&from_bob));
1751        served.unwrap();
1752        assert_eq!(synced.unwrap().rejected, 0);
1753        assert!(holds(&c, borg::msg::FILE, &line(&second)) && holds(&c, borg::msg::FILE, &line(&third)), "and carol both from bob");
1754        for n in [alice, bob, carol] {
1755            n.close().await;
1756        }
1757    }
1758
1759    /// A seen line merged from a peer moves the message log but not its talk (borg msg.rs `talk_mark`),
1760    /// so no hook or waiter on this node reads the log again for it; a tell merged moves it.
1761    #[tokio::test(flavor = "multi_thread")]
1762    async fn a_merged_seen_line_leaves_the_logs_talk_and_a_merged_tell_moves_it() {
1763        let p = crew("talk", &["alice", "bob"]);
1764        let (a, b) = (home(&p[0]), home(&p[1]));
1765        let keys: Vec<String> = [&a, &b].iter().map(|h| Identity::load_or_create(h).unwrap().public()).collect();
1766        let now = borg::learn::now_ms();
1767        let sender = |node: &str, by: &str| borg::msg::Sender { by: by.into(), name: "One of One".into(), harness: "claude".into(), node: node.into() };
1768        let x = borg::msg::Message::new(borg::msg::TELL, &sender(&keys[0], "s1"), &format!("s2@{}", keys[1]), None, now, "one");
1769        let x = borg::msg::send(&a, x, now).unwrap();
1770        let (alice, bob) = (Node::bind(&p[0], LOCAL.parse().unwrap()).await.unwrap(), Node::bind(&p[1], LOCAL.parse().unwrap()).await.unwrap());
1771        let ticket = alice.ticket();
1772        let (served, synced) = tokio::join!(alice.accept(), bob.sync(&ticket));
1773        served.unwrap();
1774        synced.unwrap();
1775        let caps = borg::harness::by_name("claude").unwrap().caps();
1776        let out = borg::msg::deliver(&b, caps, "prompt", &json!({"session_id": "s2"}), Some(now - 10), now + 1, None).expect("shown on bob's node");
1777        assert_eq!(out["shown_ids"], json!([x.id]));
1778        let (log, talk) = (borg::msg::mark_of(&borg::msg::path(&a)), borg::msg::talk_mark(&a));
1779        let (served, synced) = tokio::join!(alice.accept(), bob.sync(&ticket));
1780        served.unwrap();
1781        assert_eq!(synced.unwrap().rejected, 0);
1782        assert!(
1783            lines::read(&borg::msg::path(&a)).values().any(|l| matches!(Line::parse(l), Some(Line::Seen(s)) if s.ids == [x.id.clone()])),
1784            "the seen line merged"
1785        );
1786        assert_ne!(borg::msg::mark_of(&borg::msg::path(&a)), log, "the log moved");
1787        assert_eq!(borg::msg::talk_mark(&a), talk, "its talk did not");
1788        // Bob's s2 answers: a tell merged moves the talk, and alice's s1 is shown it.
1789        let reply = borg::msg::Message::new(borg::msg::TELL, &sender(&keys[1], "s2"), &format!("s1@{}", keys[0]), Some(&x.id), now + 2, "done");
1790        let reply = borg::msg::send(&b, reply, now + 2).unwrap();
1791        let (served, synced) = tokio::join!(alice.accept(), bob.sync(&ticket));
1792        served.unwrap();
1793        synced.unwrap();
1794        assert_ne!(borg::msg::talk_mark(&a), talk, "a tell moves it");
1795        let out = borg::msg::deliver(&a, caps, "prompt", &json!({"session_id": "s1"}), Some(now - 10), now + 3, None).expect("shown on alice's node");
1796        assert_eq!(out["shown_ids"], json!([reply.id]));
1797        for n in [alice, bob] {
1798            n.close().await;
1799        }
1800    }
1801
1802    /// A 0.4.0 or 0.5.x member (`borg/mesh/13`) is offered only what it reads and keeps, whichever side
1803    /// dials: v1 lines in 0.5.0's pacing, never a v2 text or a seen line. Its own lines reach this node.
1804    #[tokio::test(flavor = "multi_thread")]
1805    async fn a_mesh_13_peer_gets_only_v1_lines_in_v1_pacing_both_ways() {
1806        let p = crew("v13", &["alice", "old1", "old2"]);
1807        let (a, o1, o2) = (home(&p[0]), home(&p[1]), home(&p[2]));
1808        let keys: Vec<String> = [&a, &o1, &o2].iter().map(|h| Identity::load_or_create(h).unwrap().public()).collect();
1809        let now = borg::learn::now_ms();
1810        let to = format!("s-r@{}", keys[0]);
1811        let (first, first_line) = line_of(&keys[0], "s1", &to, now - 3000, "first");
1812        let seen = seen_of(&keys[0], "s-r", now - 2500, &[&first.id]);
1813        let (second, second_line) = line_of(&keys[0], "s1", &to, now - 2000, "second");
1814        let (long, long_line) = line_of(&keys[0], "s1", &format!("s-x@{}", keys[0]), now - 1000, &"w".repeat(borg::msg::V1_TEXT + 1));
1815        assert_eq!((first.v, second.v, long.v), (1, 1, 2));
1816        append_lines(&a, borg::msg::FILE, &[first_line.clone(), seen.clone(), second_line.clone(), long_line.clone()]);
1817        let theirs: Vec<String> = [(&o1, &keys[1]), (&o2, &keys[2])].iter().map(|(h, k)| message(h, k, ALL_TO, now - 100, "from an older node")).collect();
1818        let alice = Node::bind(&p[0], LOCAL.parse().unwrap()).await.unwrap();
1819        let old = |i: usize| Node::open(&p[i], Some(LOCAL.parse().unwrap()), true, |x| x.alpns(vec![V13_ALPN.to_vec()]));
1820        let (old1, old2) = (old(1).await.unwrap(), old(2).await.unwrap());
1821        // The older node dials.
1822        let ticket = alice.ticket();
1823        let (served, dialled) = tokio::join!(alice.accept(), old1.dial(&ticket, &[], V13_ALPN, Vec::new()));
1824        served.unwrap();
1825        dialled.unwrap();
1826        // This node dials the older one, offering its own protocol first.
1827        let old_ticket = old2.ticket();
1828        let (served, dialled) = tokio::join!(old2.accept(), alice.sync(&old_ticket));
1829        served.unwrap();
1830        dialled.unwrap();
1831        for o in [&o1, &o2] {
1832            assert!(holds(o, borg::msg::FILE, &first_line), "a v1 line travels");
1833            assert!(!holds(o, borg::msg::FILE, &second_line), "0.5.0's pacing: one per sender and recipient per rate slot");
1834            assert!(!holds(o, borg::msg::FILE, &long_line) && !holds(o, borg::msg::FILE, &seen), "no v2 text, no seen line");
1835            let v1 = |l: &String| serde_json::from_str::<borg::msg::Message>(l).is_ok_and(|m| m.v == 1);
1836            assert!(lines::read(&o.join(borg::msg::FILE)).values().all(v1), "every line a 0.5.0 reader reads");
1837        }
1838        assert!(theirs.iter().all(|l| holds(&a, borg::msg::FILE, l)), "their lines reach this node");
1839        for n in [alice, old1, old2] {
1840            n.close().await;
1841        }
1842    }
1843
1844    /// A session on a 0.4.0 or 0.5.x node (`borg/mesh/13`) acknowledges nothing, so this node's `send`
1845    /// paces its sender as 0.5.0 did (borg msg.rs `Pace::Rate`): two tells a rate slot apart both go,
1846    /// unread, and that node takes both.
1847    #[tokio::test(flavor = "multi_thread")]
1848    async fn two_tells_a_rate_slot_apart_reach_a_mesh_13_node() {
1849        let p = crew("v13-rate", &["alice", "old"]);
1850        let (a, o) = (home(&p[0]), home(&p[1]));
1851        let keys: Vec<String> = [&a, &o].iter().map(|h| Identity::load_or_create(h).unwrap().public()).collect();
1852        let now = borg::learn::now_ms();
1853        let from = borg::msg::Sender { by: "s1".into(), name: "One of One".into(), harness: "claude".into(), node: keys[0].clone() };
1854        let to = format!("s9@{}", keys[1]);
1855        let tell = |ms: i64, text: &str| borg::msg::send(&a, borg::msg::Message::new(borg::msg::TELL, &from, &to, None, ms, text), ms);
1856        let first = tell(now - V1_RATE_MS - 1000, "first").unwrap();
1857        assert!(matches!(tell(now - V1_RATE_MS, "too soon"), Err(borg::msg::SendError::Window { pace: borg::msg::Pace::Rate, .. })));
1858        let second = tell(now - 500, "a slot later").unwrap();
1859        let alice = Node::bind(&p[0], LOCAL.parse().unwrap()).await.unwrap();
1860        let old = Node::open(&p[1], Some(LOCAL.parse().unwrap()), true, |x| x.alpns(vec![V13_ALPN.to_vec()])).await.unwrap();
1861        let ticket = alice.ticket();
1862        let (served, dialled) = tokio::join!(alice.accept(), old.dial(&ticket, &[], V13_ALPN, Vec::new()));
1863        served.unwrap();
1864        dialled.unwrap();
1865        let line = |m: &borg::msg::Message| serde_json::to_string(m).unwrap();
1866        assert!(holds(&o, borg::msg::FILE, &line(&first)) && holds(&o, borg::msg::FILE, &line(&second)), "both offered and taken");
1867        for n in [alice, old] {
1868            n.close().await;
1869        }
1870    }
1871
1872    /// A text longer than a 0.4.0 or 0.5.x node (`borg/mesh/13`) reads is refused for its sessions at
1873    /// `send`, saying why: that node would never show it, and it would hold the sender's window for a
1874    /// week. A short text goes, so does a long one for the collective, and once the node's sessions run
1875    /// on this protocol, a long one for it too.
1876    #[tokio::test(flavor = "multi_thread")]
1877    async fn a_text_a_mesh_13_node_cannot_read_is_refused_for_its_sessions() {
1878        let p = crew("v13-long", &["alice", "old"]);
1879        let (a, o) = (home(&p[0]), home(&p[1]));
1880        let keys: Vec<String> = [&a, &o].iter().map(|h| Identity::load_or_create(h).unwrap().public()).collect();
1881        let now = borg::learn::now_ms();
1882        let from = borg::msg::Sender { by: "s1".into(), name: "One of One".into(), harness: "claude".into(), node: keys[0].clone() };
1883        let tell = |to: &str, ms: i64, text: &str| borg::msg::send(&a, borg::msg::Message::new(borg::msg::TELL, &from, to, None, ms, text), ms);
1884        let long = "w".repeat(borg::msg::V1_TEXT + 1);
1885        let alice = Node::bind(&p[0], LOCAL.parse().unwrap()).await.unwrap();
1886        let old = Node::open(&p[1], Some(LOCAL.parse().unwrap()), true, |x| x.alpns(vec![V13_ALPN.to_vec()])).await.unwrap();
1887        let ticket = alice.ticket();
1888        let (served, dialled) = tokio::join!(alice.accept(), old.dial(&ticket, &[], V13_ALPN, Vec::new()));
1889        served.unwrap();
1890        dialled.unwrap();
1891        assert_eq!(borg::presence::reads(&a).get(&keys[1]), Some(&1));
1892        let to = format!("s9@{}", keys[1]);
1893        let refused = tell(&to, now, &long).unwrap_err();
1894        assert!(matches!(&refused, borg::msg::SendError::Reads { v: 1, .. }), "{refused:?}");
1895        assert!(refused.to_string().contains(&format!("at most {} characters", borg::msg::V1_TEXT)), "{refused}");
1896        tell(&to, now + 1, "short").expect("a v1 text goes, and nothing held the window");
1897        tell(ALL_TO, now + 2, &long).expect("the other members read it");
1898        let (served, dialled) = tokio::join!(alice.accept(), old.dial(&ticket, &[], ALPN, Vec::new()));
1899        served.unwrap();
1900        dialled.unwrap();
1901        tell(&format!("s8@{}", keys[1]), now + 3, &long).expect("upgraded");
1902        alice.close().await;
1903        old.close().await;
1904    }
1905
1906    /// Two members that see a membership change at different times keep the same messages: neither the
1907    /// window nor the bound depends on how many members there are (0.5.0 shared its line bound among
1908    /// them, so the node that saw a third member first cut lines the other still offered).
1909    #[tokio::test(flavor = "multi_thread")]
1910    async fn a_membership_change_seen_at_different_times_keeps_the_messages() {
1911        let (pa, pb) = team("members");
1912        let (a, b) = (home(&pa), home(&pb));
1913        let alice_id = Identity::load_or_create(&a).unwrap();
1914        let bob_key = Identity::load_or_create(&b).unwrap().public();
1915        let now = borg::learn::now_ms();
1916        let n = V1_KEEP / 2 + 50;
1917        let sent: Vec<String> = (0..n).map(|i| message(&a, &alice_id.public(), &format!("s-{i:03}@{bob_key}"), now - (n - i) as i64, "hi")).collect();
1918        let alice = Node::bind(&pa, LOCAL.parse().unwrap()).await.unwrap();
1919        let bob = Node::bind(&pb, LOCAL.parse().unwrap()).await.unwrap();
1920        let (ticket, bob_ticket) = (alice.ticket(), bob.ticket());
1921        let (served, synced) = tokio::join!(alice.accept(), bob.sync(&ticket));
1922        served.unwrap();
1923        assert_eq!(synced.unwrap().rejected, 0);
1924        // A third member, on alice's node first: bob adopts it during the next session.
1925        let carol = Identity::load_or_create(&a.join("carol")).unwrap();
1926        let mut m = Membership::load(&a).unwrap().unwrap();
1927        m.invite(&alice_id, &carol.public(), "member").unwrap();
1928        m.save(&a).unwrap();
1929        let (served, synced) = tokio::join!(alice.accept(), bob.sync(&ticket));
1930        assert_eq!((served.unwrap().rejected, synced.unwrap().rejected), (0, 0));
1931        let (served, synced) = tokio::join!(bob.accept(), alice.sync(&bob_ticket));
1932        assert_eq!((served.unwrap().rejected, synced.unwrap().rejected), (0, 0));
1933        for h in [&a, &b] {
1934            assert!(sent.iter().all(|l| holds(h, borg::msg::FILE, l)), "every message kept");
1935        }
1936        assert_eq!(Membership::load(&b).unwrap().unwrap().members.len(), 3, "bob saw the change");
1937        alice.close().await;
1938        bob.close().await;
1939    }
1940
1941    #[tokio::test(flavor = "multi_thread")]
1942    async fn team_outcomes_sync_and_count_once() {
1943        let (pa, pb) = team("outcomes");
1944        let (a, b) = (home(&pa), home(&pb));
1945        write(&a, "lessons.jsonl", &[lesson("L1", 1, "lock order?")]);
1946        let o = |kind: &str, ms| json!({"kind": kind, "session": "s1", "lessons": ["L1"], "ms": ms, "repo": "app", "groups": [["L1", "L2"]]}).to_string();
1947        write(&a, "outcomes.jsonl", &[o("asked", 1), o("asked_then_changed", 2)]);
1948        assert_eq!(borg::outcome::publish(&a), 2);
1949        let alice = Node::bind(&pa, LOCAL.parse().unwrap()).await.unwrap();
1950        let bob = Node::bind(&pb, LOCAL.parse().unwrap()).await.unwrap();
1951        let ticket = alice.ticket();
1952        let (served, synced) = tokio::join!(alice.accept(), bob.sync(&ticket));
1953        served.unwrap();
1954        synced.unwrap();
1955        let team = borg::outcome::TEAM_FILE;
1956        assert_eq!(lines::read(&b.join(team)), lines::read(&a.join(team)), "bob holds alice's outcomes");
1957        let text = std::fs::read_to_string(b.join(team)).unwrap();
1958        assert!(!text.contains("\"s1\"") && !text.contains("app") && !text.contains("groups"), "redacted: {text}");
1959        assert!(!b.join("outcomes.jsonl").exists(), "the raw log stays with alice");
1960        let l1 = [borg::conscience::Challenge { id: "L1".into(), ..Default::default() }];
1961        let changed = |h: &Path| borg::memory::derive(&borg::outcome::read_team(&h.join("lessons.jsonl")), &l1, None)["L1"].changed;
1962        assert_eq!(changed(&b), 1, "alice's session counts on bob");
1963        assert_eq!(changed(&a), 1, "alice's own published copy does not count twice");
1964        let (served, synced) = tokio::join!(alice.accept(), bob.sync(&ticket));
1965        let quiet = Report { rounds: SYNCED.len(), fps: SYNCED.len(), ..Report::default() };
1966        assert_eq!((synced.unwrap(), served.unwrap()), (quiet, quiet), "resync is quiet");
1967        alice.close().await;
1968        bob.close().await;
1969    }
1970
1971    #[tokio::test(flavor = "multi_thread")]
1972    async fn two_members_end_with_the_union_and_resync_moves_only_digests() {
1973        let (pa, pb) = team("union");
1974        let (a, b) = (home(&pa), home(&pb));
1975        write(&a, "lessons.jsonl", &[lesson("L1", 1, "lock order?"), lesson("L2", 1, "retry budget?")]);
1976        write(&b, "lessons.jsonl", &[lesson("L1", 1, "lock order?"), lesson("L1", 2, "lock order, also in tests?"), lesson("L3", 1, "idempotent?")]);
1977
1978        // Since bob joined, alice invited carol: bob learns of her through the sync (trust first).
1979        let carol = Identity::load_or_create(&home(&node("union-carol", "team"))).unwrap().public();
1980        let mut m = Membership::load(&a).unwrap().unwrap();
1981        m.invite(&Identity::load_or_create(&a).unwrap(), &carol, "member").unwrap();
1982        m.save(&a).unwrap();
1983
1984        let alice = Node::bind(&pa, LOCAL.parse().unwrap()).await.unwrap();
1985        let bob = Node::bind(&pb, LOCAL.parse().unwrap()).await.unwrap();
1986        let ticket = alice.ticket();
1987        let (served, synced) = tokio::join!(alice.accept(), bob.sync(&ticket));
1988        let (served, synced) = (served.unwrap(), synced.unwrap());
1989        assert_eq!((synced.received, synced.sent), (1, 2), "bob gets L2, gives L1v2 and L3");
1990        assert_eq!((served.received, served.sent), (2, 1));
1991        assert_eq!(Membership::load(&b).unwrap().unwrap().role_of(&carol), Some("member"), "bob adopted the newer membership");
1992        for file in SYNCED {
1993            assert_eq!(lines::read(&a.join(file)), lines::read(&b.join(file)), "{file}: both hold the union");
1994        }
1995        let asked = |h: &Path| {
1996            let mut v: Vec<(String, String)> = borg::conscience::load(&h.join("lessons.jsonl"), None).into_iter().map(|c| (c.id, c.question)).collect();
1997            v.sort();
1998            v
1999        };
2000        assert_eq!(asked(&a), asked(&b));
2001        assert!(asked(&a).contains(&("L1".into(), "lock order, also in tests?".into())), "the newer version wins on alice too");
2002
2003        let (served, synced) = tokio::join!(alice.accept(), bob.sync(&ticket));
2004        // One round and one fingerprint per synced log (lessons, stats, outcomes), nothing else.
2005        let quiet = Report { rounds: SYNCED.len(), fps: SYNCED.len(), ..Report::default() };
2006        assert_eq!(synced.unwrap(), quiet, "re-sync: one fingerprint per log, nothing new");
2007        assert_eq!(served.unwrap(), quiet);
2008        alice.close().await;
2009        bob.close().await;
2010    }
2011
2012    /// Each episode is learned once, by the node that captured it: before alice's lesson arrives
2013    /// bob has nothing to learn, afterwards he holds her lesson and still asks no model.
2014    #[tokio::test(flavor = "multi_thread")]
2015    async fn an_episode_is_learned_only_where_it_was_captured() {
2016        let (pa, pb) = team("episodes");
2017        let (a, b) = (home(&pa), home(&pb));
2018        let e1 = json!({"id": "e1", "ms": 1, "repo": "r", "session": "s", "task": "t", "agent": "a",
2019            "rejected_tool": null, "correction": "check TOCTOU on the flag", "engineer": "ann@x"});
2020        write(&a, "episodes.jsonl", &[e1.to_string()]);
2021        let calls = std::cell::Cell::new(0);
2022        let ask = |_: &str| {
2023            calls.set(calls.get() + 1);
2024            Some(json!({"is_lesson": true, "question": "Is the check-then-act atomic?", "situations": [], "consequence": "race", "same_as": ""}))
2025        };
2026
2027        let alice = Node::bind(&pa, LOCAL.parse().unwrap()).await.unwrap();
2028        let bob = Node::bind(&pb, LOCAL.parse().unwrap()).await.unwrap();
2029        let ticket = alice.ticket();
2030        let (served, synced) = tokio::join!(alice.accept(), bob.sync(&ticket));
2031        served.unwrap();
2032        synced.unwrap();
2033        assert!(!b.join("episodes.jsonl").exists(), "alice's raw episode stays with alice");
2034        assert_eq!(borg::learn::learn_in(&b, None, &ask, false, 2), (0, 0, 0));
2035        assert_eq!(calls.get(), 0, "bob asks no model for alice's episode");
2036
2037        assert_eq!(borg::learn::learn_in(&a, None, &ask, false, 2), (1, 0, 0));
2038        assert_eq!(calls.get(), 1, "alice learns e1 once");
2039        let (served, synced) = tokio::join!(alice.accept(), bob.sync(&ticket));
2040        served.unwrap();
2041        assert_eq!(synced.unwrap().received, 1, "bob receives the lesson");
2042        let got = borg::learn::lessons(&b.join("lessons.jsonl"));
2043        assert_eq!(
2044            (got.len(), got[0].id.as_str(), got[0].episodes.clone(), got[0].engineers.clone()),
2045            (1, "Le1", vec!["e1".to_string()], vec!["ann@x".to_string()])
2046        );
2047        assert_eq!(borg::learn::learn_in(&b, None, &ask, false, 3), (0, 0, 0));
2048        assert_eq!(calls.get(), 1, "and still learns nothing again");
2049        alice.close().await;
2050        bob.close().await;
2051    }
2052
2053    /// A member on an older protocol (a node that only speaks `borg/mesh/5`) is told apart from a broken
2054    /// network both ways: the session ends at once, the error and the record (for `borg doctor`) name
2055    /// the peer and who has to upgrade, and the first session that completes drops the record.
2056    #[tokio::test(flavor = "multi_thread")]
2057    async fn a_member_on_an_older_protocol_is_named_both_ways() {
2058        const OLD: &[u8] = b"borg/mesh/5";
2059        let (pa, pb) = team("protocol");
2060        let (a, b) = (home(&pa), home(&pb));
2061        let bob_key = Identity::load_or_create(&b).unwrap().public();
2062        let alice = Node::bind(&pa, LOCAL.parse().unwrap()).await.unwrap();
2063        let old_bob = Node::open(&pb, Some(LOCAL.parse().unwrap()), true, |x| x.alpns(vec![OLD.to_vec()])).await.unwrap();
2064
2065        // Alice dials the old node.
2066        let serve_old = async {
2067            let conn = old_bob.incoming().await?.await.ok()?;
2068            Some((conn.alpn().to_vec(), conn.closed().await.to_string()))
2069        };
2070        let old_ticket = old_bob.ticket();
2071        let (seen, synced) = tokio::join!(serve_old, alice.sync(&old_ticket));
2072        let err = synced.unwrap_err();
2073        let ours = String::from_utf8_lossy(ALPN);
2074        let line = format!("peer {} speaks borg/mesh/5, this node {ours}: nothing syncs with it; upgrade it", &bob_key[..12]);
2075        assert_eq!(err, line);
2076        let (alpn, closed) = seen.expect("the old node completed the handshake");
2077        assert_eq!(alpn, OLD);
2078        let alice_key = Identity::load_or_create(&a).unwrap().public();
2079        let told = format!("peer {} speaks {ours}, this node borg/mesh/5: nothing syncs with it; upgrade this node", &alice_key[..12]);
2080        assert!(closed.contains(&told), "the old node is told why, as it reads it: {closed}");
2081        assert_eq!(borg::presence::mismatches(&a)[&bob_key].theirs, "borg/mesh/5");
2082
2083        // The old node dials alice.
2084        let _ = std::fs::remove_file(borg::presence::protocols_path(&a));
2085        let addr = Target::parse(&alice.ticket()).unwrap().endpoint_addr(&old_bob.relay).unwrap();
2086        let dial_old = async {
2087            let conn = old_bob.endpoint.connect(addr, OLD).await.map_err(|e| e.to_string())?;
2088            Ok::<String, String>(conn.closed().await.to_string())
2089        };
2090        let (served, dialled) = tokio::join!(alice.accept(), dial_old);
2091        assert_eq!(served.unwrap_err(), line);
2092        assert!(dialled.unwrap().contains(&told));
2093        assert_eq!(borg::presence::mismatches(&a)[&bob_key].ours, ours);
2094        old_bob.close().await;
2095
2096        // Bob upgraded: the first session that completes drops the record.
2097        let bob = Node::bind(&pb, LOCAL.parse().unwrap()).await.unwrap();
2098        let ticket = alice.ticket();
2099        let (served, synced) = tokio::join!(alice.accept(), bob.sync(&ticket));
2100        served.unwrap();
2101        synced.unwrap();
2102        assert!(borg::presence::mismatches(&a).is_empty());
2103        alice.close().await;
2104        bob.close().await;
2105    }
2106
2107    #[test]
2108    fn older_protocols_are_every_version_below_the_spoken_ones() {
2109        let n = version(LEGACY_ALPN).expect("borg/mesh/<n>");
2110        assert_eq!((version(V13_ALPN), version(ALPN)), (Some(n + 1), Some(n + 2)), "this protocol, 0.4.0's and 0.3.0's are the three spoken");
2111        assert!(spoken(ALPN) && spoken(V13_ALPN) && spoken(LEGACY_ALPN));
2112        let o = older();
2113        let below = format!("borg/mesh/{}", n - 1);
2114        assert_eq!((o.len(), o[0].as_slice(), o[o.len() - 1].as_slice()), (n as usize - 1, below.as_bytes(), b"borg/mesh/1".as_slice()));
2115        assert!(o.iter().all(|a| !spoken(a)));
2116        assert_eq!(synced(ALPN).filter(|f| *f != PLAYBOOKS).collect::<Vec<_>>(), SYNCED, "every synced log, in order");
2117        assert_eq!(synced(ALPN).collect::<Vec<_>>(), [&OLD_SYNCED[..], &[borg::msg::FILE]].concat(), "0.4.0's list, in its order");
2118        assert_eq!(synced(V13_ALPN).collect::<Vec<_>>(), synced(ALPN).collect::<Vec<_>>(), "0.4.0's list is this one");
2119        assert_eq!(synced(LEGACY_ALPN).collect::<Vec<_>>(), OLD_SYNCED, "0.3.0's list, in its order");
2120    }
2121
2122    #[tokio::test(flavor = "multi_thread")]
2123    async fn an_unlisted_key_is_rejected_both_ways() {
2124        let (pa, pb) = team("unlisted");
2125        let a = home(&pa);
2126        write(&a, "lessons.jsonl", &[lesson("L1", 1, "secret sauce?")]);
2127        // Eve forges her own "work" membership that lists alice, but alice's does not list eve.
2128        let pe = node("unlisted-eve", "team");
2129        let e = home(&pe);
2130        let eve = Identity::load_or_create(&e).unwrap();
2131        let mut forged = Membership::found("work", &eve);
2132        forged.invite(&eve, &Identity::load_or_create(&a).unwrap().public(), "member").unwrap();
2133        forged.save(&e).unwrap();
2134
2135        let alice = Node::bind(&pa, LOCAL.parse().unwrap()).await.unwrap();
2136        let eve_node = Node::bind(&pe, LOCAL.parse().unwrap()).await.unwrap();
2137        let ticket = alice.ticket();
2138        let (served, synced) = tokio::join!(alice.accept(), eve_node.sync(&ticket));
2139        assert!(served.unwrap_err().contains("not a member"), "alice turns eve away");
2140        assert!(synced.is_err());
2141        assert!(!e.join("lessons.jsonl").exists(), "eve received nothing");
2142
2143        // And a member never dials a key its membership does not list.
2144        let bob = Node::bind(&pb, LOCAL.parse().unwrap()).await.unwrap();
2145        let err = bob.sync(&eve_node.ticket()).await.unwrap_err();
2146        assert!(err.contains("not a member") && err.contains("refusing to connect"), "{err}");
2147        for n in [alice, eve_node, bob] {
2148            n.close().await;
2149        }
2150    }
2151
2152    #[tokio::test(flavor = "multi_thread")]
2153    async fn a_local_collective_or_off_repo_never_syncs() {
2154        let (pa, _) = team("local");
2155        write(&home(&pa), "lessons.jsonl", &[lesson("L1", 1, "private thought?")]);
2156        configure(&pa, "local");
2157        assert!(!collective::may_sync(&pa.route()));
2158        let err = Node::bind(&pa, LOCAL.parse().unwrap()).await.err().expect("local must be refused");
2159        assert!(err.contains("is local") && err.contains("refusing to sync"), "{err}");
2160        let off = Place { dir: "/somewhere/else".into(), ..pa.clone() };
2161        let err = Node::bind(&off, LOCAL.parse().unwrap()).await.err().expect("off must be refused");
2162        assert!(err.contains("off"), "{err}");
2163        std::fs::write(&pa.config, "{ broken").unwrap();
2164        assert_eq!(pa.route(), Route::Off, "a broken config routes off");
2165        assert!(Node::bind(&pa, LOCAL.parse().unwrap()).await.is_err());
2166        let team = Route::Collective { name: "work".into(), home: home(&pa), share: collective::Share::Team };
2167        assert!(gate(&team).is_ok() && gate(&Route::Off).is_err());
2168    }
2169
2170    /// A running `serve` re-reads the config at each session: once the collective is set to
2171    /// `share: local`, the next peer is refused before a single frame of it is sent.
2172    #[tokio::test(flavor = "multi_thread")]
2173    async fn serve_stops_syncing_when_the_collective_turns_local() {
2174        let (pa, pb) = team("turns-local");
2175        write(&home(&pa), "lessons.jsonl", &[lesson("L1", 1, "now private?")]);
2176        let alice = Node::bind(&pa, LOCAL.parse().unwrap()).await.unwrap();
2177        let bob = Node::bind(&pb, LOCAL.parse().unwrap()).await.unwrap();
2178        let ticket = alice.ticket();
2179        configure(&pa, "local");
2180        let (served, synced) = tokio::join!(alice.accept(), bob.sync(&ticket));
2181        let err = served.unwrap_err();
2182        assert!(err.contains("is local") && err.contains("refusing to sync"), "{err}");
2183        assert!(synced.is_err());
2184        assert!(!home(&pb).join("lessons.jsonl").exists(), "bob received nothing");
2185        // A dialling node is gated the same way.
2186        configure(&pb, "local");
2187        assert!(bob.sync(&ticket).await.unwrap_err().contains("is local"));
2188        alice.close().await;
2189        bob.close().await;
2190    }
2191
2192    /// Second guard: a team route whose store a local collective also maps to never syncs, even if
2193    /// such a config got past `Config::check`.
2194    #[test]
2195    fn a_store_shared_with_a_local_collective_is_refused() {
2196        let p = node("shared-store", "team");
2197        let cfg: Config = serde_json::from_value(json!({"collectives": [
2198            {"name": "work", "paths": [p.dir], "share": "team"},
2199            {"name": "work", "paths": ["/elsewhere"], "share": "local"}]}))
2200        .unwrap();
2201        let route = collective::route(Some(&cfg), &p.dir, &p.base);
2202        assert!(gate(&route).is_ok(), "the route alone looks shareable");
2203        assert_eq!(collective::local_owner_of(Some(&cfg), &home(&p), &p.base), Some("work"));
2204        // Written to disk, load rejects the config outright, which routes off.
2205        std::fs::write(
2206            &p.config,
2207            serde_json::to_string(&json!({"collectives": [
2208            {"name": "work", "paths": [p.dir], "share": "team"},
2209            {"name": "work", "paths": ["/elsewhere"], "share": "local"}]}))
2210            .unwrap(),
2211        )
2212        .unwrap();
2213        assert!(p.gate().is_err());
2214    }
2215
2216    /// Alice founds the collective and invites everyone else; all install the same document.
2217    fn crew(tag: &str, names: &[&str]) -> Vec<Place> {
2218        let places: Vec<Place> = names.iter().map(|n| node(&format!("{tag}-{n}"), "team")).collect();
2219        let ids: Vec<Identity> = places.iter().map(|p| Identity::load_or_create(&home(p)).unwrap()).collect();
2220        let mut m = Membership::found("work", &ids[0]);
2221        for id in &ids[1..] {
2222            m.invite(&ids[0], &id.public(), "member").unwrap();
2223        }
2224        for p in &places {
2225            m.save(&home(p)).unwrap();
2226        }
2227        places
2228    }
2229
2230    fn append_lines(home: &Path, file: &str, lines: &[String]) {
2231        use std::io::Write;
2232        let mut f = std::fs::OpenOptions::new().create(true).append(true).open(home.join(file)).unwrap();
2233        for l in lines {
2234            writeln!(f, "{l}").unwrap();
2235        }
2236    }
2237
2238    fn holds(home: &Path, file: &str, line: &str) -> bool {
2239        lines::read(&home.join(file)).contains_key(&lines::hash(line))
2240    }
2241
2242    async fn eventually(what: &str, limit: std::time::Duration, f: impl Fn() -> bool) {
2243        let start = std::time::Instant::now();
2244        while !f() {
2245            assert!(start.elapsed() < limit, "timed out waiting for: {what}");
2246            tokio::time::sleep(std::time::Duration::from_millis(25)).await;
2247        }
2248    }
2249
2250    fn addr_of(n: &Node) -> String {
2251        n.ticket().split_once('@').unwrap().1.to_string()
2252    }
2253
2254    /// Two `serve` loops: a lesson either side writes reaches the other with no manual sync, and
2255    /// once alice's collective turns local, her next lesson stays home. Both stop cleanly.
2256    #[tokio::test(flavor = "multi_thread")]
2257    async fn serve_spreads_new_lessons_without_a_manual_sync() {
2258        use crate::daemon::{self, Event, Options};
2259        use std::sync::Arc;
2260        use std::time::Duration;
2261        let p = crew("daemon", &["alice", "bob"]);
2262        let (a, b) = (home(&p[0]), home(&p[1]));
2263        let alice = Arc::new(Node::bind(&p[0], LOCAL.parse().unwrap()).await.unwrap());
2264        let bob = Arc::new(Node::bind(&p[1], LOCAL.parse().unwrap()).await.unwrap());
2265        // Bob knows where alice is; alice learns bob's address when he first dials in.
2266        assert!(peers::remember(&b, &alice.public(), &addr_of(&alice)).unwrap());
2267        let opts = Options {
2268            interval: Duration::from_millis(300),
2269            poll: Duration::from_millis(50),
2270            backoff_max: Duration::from_millis(600),
2271            timeout: Duration::from_secs(5),
2272            grace: Duration::from_secs(1),
2273        };
2274        let synced = Arc::new(std::sync::atomic::AtomicUsize::new(0));
2275        let mut stops = Vec::new();
2276        let mut loops = Vec::new();
2277        for n in [alice.clone(), bob.clone()] {
2278            let (tx, rx) = tokio::sync::oneshot::channel::<()>();
2279            stops.push(tx);
2280            let count = synced.clone();
2281            loops.push(tokio::spawn(daemon::serve(
2282                n,
2283                opts,
2284                async move {
2285                    let _ = rx.await;
2286                },
2287                move |e| {
2288                    if matches!(e, Event::Synced { result: Ok(_), .. }) {
2289                        count.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
2290                    }
2291                },
2292            )));
2293        }
2294        let l1 = lesson("L1", 1, "who owns the lock?");
2295        append_lines(&a, "lessons.jsonl", std::slice::from_ref(&l1));
2296        eventually("alice's lesson at bob", Duration::from_secs(10), || holds(&b, "lessons.jsonl", &l1)).await;
2297        let l2 = lesson("L2", 1, "is the retry idempotent?");
2298        append_lines(&b, "lessons.jsonl", std::slice::from_ref(&l2));
2299        eventually("bob's lesson at alice", Duration::from_secs(10), || holds(&a, "lessons.jsonl", &l2)).await;
2300        assert_eq!(peers::load(&a).get(&bob.public()), Some(&addr_of(&bob)), "alice learned bob's address from his hello");
2301        // The receiver holds a lesson once its served session wrote it; the dialler's round reports
2302        // `Synced` only after it closes, a moment later.
2303        eventually("a successful sync round", Duration::from_secs(10), || synced.load(std::sync::atomic::Ordering::Relaxed) > 0).await;
2304
2305        // Alice's collective turns local: her loops stop dialling, bob is turned away.
2306        configure(&p[0], "local");
2307        let l3 = lesson("L3", 1, "private now?");
2308        append_lines(&a, "lessons.jsonl", std::slice::from_ref(&l3));
2309        tokio::time::sleep(Duration::from_millis(1500)).await;
2310        assert!(!holds(&b, "lessons.jsonl", &l3), "a local collective's lesson never leaves");
2311
2312        for s in stops {
2313            let _ = s.send(());
2314        }
2315        for l in loops {
2316            tokio::time::timeout(Duration::from_secs(5), l).await.expect("serve stops within its grace").unwrap();
2317        }
2318    }
2319
2320    /// `serve` loops for `nodes`, each stopped by its sender; the failures they report, with who and when.
2321    struct Serving {
2322        stops: Vec<tokio::sync::oneshot::Sender<()>>,
2323        loops: Vec<tokio::task::JoinHandle<()>>,
2324        failed: std::sync::Arc<std::sync::Mutex<Vec<String>>>,
2325    }
2326
2327    fn serving(nodes: Vec<(&'static str, std::sync::Arc<Node>)>, opts: crate::daemon::Options) -> Serving {
2328        let s = Serving { stops: Vec::new(), loops: Vec::new(), failed: Default::default() };
2329        nodes.into_iter().fold(s, |mut s, (who, n)| {
2330            s.add(who, n, opts);
2331            s
2332        })
2333    }
2334
2335    impl Serving {
2336        fn add(&mut self, who: &'static str, n: std::sync::Arc<Node>, opts: crate::daemon::Options) {
2337            use crate::daemon::{self, Event};
2338            let (tx, rx) = tokio::sync::oneshot::channel::<()>();
2339            self.stops.push(tx);
2340            let failed = self.failed.clone();
2341            self.loops.push(tokio::spawn(daemon::serve(
2342                n,
2343                opts,
2344                async move {
2345                    let _ = rx.await;
2346                },
2347                move |e| {
2348                    if matches!(e, Event::Served(Err(_)) | Event::Synced { result: Err(_), .. } | Event::Gate(Err(_))) {
2349                        failed.lock().unwrap().push(format!("{who} at {}: {e:?}", borg::learn::now_ms()));
2350                    }
2351                },
2352            )));
2353        }
2354
2355        async fn stop(self) {
2356            for s in self.stops {
2357                let _ = s.send(());
2358            }
2359            for l in self.loops {
2360                tokio::time::timeout(std::time::Duration::from_secs(10), l).await.expect("serve stops within its grace").unwrap();
2361            }
2362        }
2363    }
2364
2365    /// Carol is a member alice dials, and she takes the dial and never answers: she reads alice's hello
2366    /// and holds the connection until the test lets her go. While alice's dial to carol is still open,
2367    /// a line alice writes reaches bob, whom only alice's dials reach. Once carol closes, alice's dial
2368    /// fails and carol's session record from before goes with it. Ordered by events, not by time:
2369    /// the dial timeout is far past the test.
2370    #[tokio::test(flavor = "multi_thread")]
2371    async fn a_peer_that_does_not_answer_holds_no_line_back() {
2372        use std::sync::Arc;
2373        use std::time::Duration;
2374        use tokio::sync::oneshot;
2375        let p = crew("deaf", &["alice", "bob", "carol"]);
2376        let (a, b) = (home(&p[0]), home(&p[1]));
2377        let alice = Arc::new(Node::bind(&p[0], LOCAL.parse().unwrap()).await.unwrap());
2378        let bob = Arc::new(Node::bind(&p[1], LOCAL.parse().unwrap()).await.unwrap());
2379        let carol = Node::bind(&p[2], LOCAL.parse().unwrap()).await.unwrap();
2380        let key = carol.public();
2381        assert!(peers::remember(&a, &key, &addr_of(&carol)).unwrap());
2382        assert!(peers::remember(&a, &bob.public(), &addr_of(&bob)).unwrap());
2383        borg::presence::note_synced(&a, &key, borg::learn::now_ms()).unwrap();
2384        let (held_tx, held) = oneshot::channel::<()>();
2385        let (release, release_rx) = oneshot::channel::<()>();
2386        let deaf = tokio::spawn(async move {
2387            let conn = carol.incoming().await.expect("alice dials carol").await.expect("handshake");
2388            let (_tx, mut rx) = conn.accept_bi().await.expect("alice's stream");
2389            assert!(matches!(recv(&mut rx).await, Ok(Msg::Hello { .. })), "alice's hello");
2390            let _ = held_tx.send(());
2391            let _ = release_rx.await;
2392            let open = conn.close_reason().is_none();
2393            conn.close(1u32.into(), b"deaf");
2394            carol.close().await;
2395            open
2396        });
2397        let opts =
2398            crate::daemon::Options { poll: Duration::from_millis(100), timeout: Duration::from_secs(600), grace: Duration::from_secs(1), ..Default::default() };
2399        let serve = serving(vec![("alice", alice.clone()), ("bob", bob.clone())], opts);
2400        let limit = Duration::from_secs(60); // liveness only: every wait below ends on an event
2401        tokio::time::timeout(limit, held).await.expect("carol holds alice's dial").unwrap();
2402
2403        let l = rollup("n1", 1, 1);
2404        append_lines(&a, "stats.jsonl", std::slice::from_ref(&l));
2405        eventually("alice's line at bob", limit, || holds(&b, "stats.jsonl", &l)).await;
2406        assert!(serve.failed.lock().unwrap().iter().all(|f| !f.contains(&key)), "carol's dial has not ended");
2407        release.send(()).unwrap();
2408        assert!(deaf.await.unwrap(), "alice's dial to carol was still open when bob got the line");
2409
2410        eventually("alice's dial to carol failed", limit, || serve.failed.lock().unwrap().iter().any(|f| f.contains(&key))).await;
2411        eventually("carol's record dropped", limit, || !borg::presence::synced(&a).contains_key(&key)).await;
2412        serve.stop().await;
2413    }
2414
2415    /// Alice's first dial to bob fails (his node runs no `serve` yet); the lesson it was to carry reaches
2416    /// him a `poll` later, with no new change and bob never dialling in: not after a 2-minute backoff.
2417    #[tokio::test(flavor = "multi_thread")]
2418    async fn a_failed_dial_is_retried_soon_with_what_it_missed() {
2419        use std::sync::Arc;
2420        use std::time::Duration;
2421        let p = crew("retry", &["alice", "bob"]);
2422        let (a, b) = (home(&p[0]), home(&p[1]));
2423        let alice = Arc::new(Node::bind(&p[0], LOCAL.parse().unwrap()).await.unwrap());
2424        let bob = Arc::new(Node::bind(&p[1], LOCAL.parse().unwrap()).await.unwrap());
2425        assert!(peers::remember(&a, &bob.public(), &addr_of(&bob)).unwrap());
2426        let l1 = lesson("L1", 1, "who owns the lock?");
2427        append_lines(&a, "lessons.jsonl", std::slice::from_ref(&l1));
2428        let opts = crate::daemon::Options { timeout: Duration::from_secs(1), grace: Duration::from_secs(1), ..Default::default() };
2429        let mut serve = serving(vec![("alice", alice.clone())], opts);
2430        eventually("alice's dial to bob failed", Duration::from_secs(10), || !serve.failed.lock().unwrap().is_empty()).await;
2431        serve.add("bob", bob.clone(), opts);
2432        eventually("alice's lesson at bob", Duration::from_secs(10), || holds(&b, "lessons.jsonl", &l1)).await;
2433        serve.stop().await;
2434    }
2435
2436    /// A 0.3.0 peer's goal ids (3b) count once each and only when valid, so both sides follow the
2437    /// same number of reconciliations: "../evil" or "A B" never names a goal.
2438    #[test]
2439    fn legacy_goals_count_the_valid_distinct_ids() {
2440        let ids = ["g1", "../evil", "A B", "g0", "g1", "/abs", ""].map(String::from).to_vec();
2441        assert_eq!(legacy_goals(ids).into_iter().collect::<Vec<_>>(), ["g0", "g1"], "sorted, as the peer walks them");
2442        assert!(legacy_goals(vec!["x".repeat(41)]).is_empty());
2443    }
2444
2445    /// The logs a 0.3.0 node reconciled, in its order (base 188c262 lines.rs `SYNCED`).
2446    const OLD_SYNCED: [&str; 5] = ["lessons.jsonl", "stats.jsonl", "outcomes-team.jsonl", "playbooks.jsonl", "constraints.jsonl"];
2447
2448    /// A 0.3.0 node's hive goal: its seven synced logs, held in memory, each line signed by the node.
2449    struct OldGoal {
2450        logs: Vec<BTreeMap<Hash, String>>,
2451        origins: HashMap<Hash, Origin>,
2452    }
2453
2454    impl OldGoal {
2455        fn new(me: &Identity) -> OldGoal {
2456            let mut g = OldGoal { logs: vec![BTreeMap::new(); LEGACY_GOAL_LOGS], origins: HashMap::new() };
2457            for log in 0..LEGACY_GOAL_LOGS {
2458                g.add(me, log, &json!({"log": log, "n": 0}).to_string());
2459            }
2460            g
2461        }
2462
2463        fn add(&mut self, me: &Identity, log: usize, line: &str) {
2464            let h = lines::hash(line);
2465            self.origins.insert(h, Origin::new(me, &h, 1));
2466            self.logs[log].insert(h, line.into());
2467        }
2468    }
2469
2470    async fn send_raw(tx: &mut SendStream, v: &serde_json::Value) -> Result<(), String> {
2471        let body = serde_json::to_vec(v).map_err(|e| e.to_string())?;
2472        tx.write_all(&(body.len() as u32).to_be_bytes()).await.map_err(|e| e.to_string())?;
2473        tx.write_all(&body).await.map_err(|e| e.to_string())
2474    }
2475
2476    /// What a 0.3.0 node said after the hello, step by step (base 188c262 `exchange`): its goal ids
2477    /// (a duplicate and an invalid one among them), a full reconciliation of each valid goal's logs,
2478    /// and `Live` with its hive beats and a signed directive. Returns the hive lines it gave and the
2479    /// presence it took in.
2480    async fn old_exchange(n: &Node, dialler: bool, tx: &mut SendStream, rx: &mut RecvStream, goal: &OldGoal) -> Result<(usize, usize), String> {
2481        let home = n.home.as_path();
2482        let mine = members(home)?;
2483        let Msg::Membership { .. } = swap(dialler, tx, rx, Msg::Membership { docs: Membership::history(home) }).await? else { return Err("membership".into()) };
2484        n.stamp()?;
2485        let (origins, mut report) = (origin::load(home), Report::default());
2486        for file in OLD_SYNCED {
2487            n.reconcile(dialler, tx, rx, &home.join(file), &origins, &mine, false, &mut report).await?;
2488        }
2489        let ids = ["g1", "g1", "Bad Id"].map(String::from).to_vec();
2490        let Msg::Goals { ids: theirs } = swap(dialler, tx, rx, Msg::Goals { ids }).await? else { return Err("goals".into()) };
2491        assert!(theirs.is_empty(), "a new node names no goal");
2492        let mut gave = 0;
2493        for set in &goal.logs {
2494            let mut r = Recon::new(set);
2495            loop {
2496                let m = r.message();
2497                let Msg::Round { round } = swap(dialler, tx, rx, Msg::Round { round: m.clone() }).await? else { return Err("round".into()) };
2498                if r.absorb(&m, round)? {
2499                    break;
2500                }
2501            }
2502            let give = within_budget(signed(set, &goal.origins, r.give.iter()));
2503            gave += give.len();
2504            let Msg::Lines { lines } = swap(dialler, tx, rx, Msg::Lines { lines: give }).await? else { return Err("lines".into()) };
2505            assert!(lines.is_empty(), "a new node gives no hive line");
2506        }
2507        for m in [
2508            Msg::Offer { hashes: vec![] },
2509            Msg::Want { hashes: vec![] },
2510            Msg::Lines { lines: vec![] },
2511            Msg::Ask { commits: vec![] },
2512            Msg::Lines { lines: vec![] },
2513        ] {
2514            swap(dialler, tx, rx, m).await?;
2515        }
2516        let now = borg::learn::now_ms();
2517        let directive = json!({"id": "d1", "collective": "work", "key": n.me.public(), "engineer": "old@x", "target": {"node": "", "session": ""},
2518            "kind": "pause", "text": "", "ms": now, "sig": "00"});
2519        let live = json!({"t": "live", "presence": live::outgoing(home, &n.me, now), "directives": [directive],
2520            "beats": [{"goal": "g1", "line": "{}", "key": n.me.public(), "sig": "00"}]});
2521        let theirs = if dialler {
2522            send_raw(tx, &live).await?;
2523            recv(rx).await?
2524        } else {
2525            let theirs = recv(rx).await?;
2526            send_raw(tx, &live).await?;
2527            theirs
2528        };
2529        let Msg::Live { presence, .. } = theirs else { return Err("live".into()) };
2530        Ok((gave, live::receive(home, &mine, &n.me.public(), presence, now)?))
2531    }
2532
2533    async fn old_dial(n: &Node, target: &str, goal: &OldGoal) -> Result<(usize, usize), String> {
2534        let t = Target::parse(target)?;
2535        let conn = n.endpoint.connect(t.endpoint_addr(&n.relay)?, LEGACY_ALPN).await.map_err(|e| e.to_string())?;
2536        let (mut tx, mut rx) = conn.open_bi().await.map_err(|e| e.to_string())?;
2537        swap(true, &mut tx, &mut rx, Msg::Hello { collective: "work".into(), listen: None, relay: None }).await?;
2538        let got = old_exchange(n, true, &mut tx, &mut rx, goal).await;
2539        let _ = tx.finish();
2540        conn.close(0u32.into(), b"done");
2541        got
2542    }
2543
2544    async fn old_serve(n: &Node, goal: &OldGoal) -> Result<(usize, usize), String> {
2545        let conn = n.incoming().await.ok_or("closed")?.await.map_err(|e| e.to_string())?;
2546        let (mut tx, mut rx) = conn.accept_bi().await.map_err(|e| e.to_string())?;
2547        let Msg::Hello { .. } = recv(&mut rx).await? else { return Err("hello".into()) };
2548        send(&mut tx, &Msg::Hello { collective: "work".into(), listen: None, relay: None }).await?;
2549        let got = old_exchange(n, false, &mut tx, &mut rx, goal).await;
2550        let _ = tx.finish();
2551        conn.closed().await;
2552        got
2553    }
2554
2555    /// A 0.3.0 peer (it names hive goals, reconciles their logs and the playbooks, and sends beats in
2556    /// `Live`) keeps syncing with this node in both roles: lessons and presence arrive both ways, its
2557    /// directives are ignored, no hive or playbooks file is written, and a hive line it gave once is not
2558    /// sent again when this node dials.
2559    #[tokio::test(flavor = "multi_thread")]
2560    async fn a_0_3_0_peer_syncs_in_both_roles_and_its_hive_lines_come_once() {
2561        let (pa, pb) = team("legacy");
2562        let (a, b) = (home(&pa), home(&pb));
2563        let (alice_id, bob_id) = (Identity::load_or_create(&a).unwrap(), Identity::load_or_create(&b).unwrap());
2564        let mut m = Membership::load(&a).unwrap().unwrap();
2565        m.invite(&alice_id, &bob_id.public(), "steer").unwrap();
2566        m.save(&a).unwrap();
2567        m.save(&b).unwrap();
2568        let (l1, l2) = (lesson("L1", 1, "lock order?"), lesson("L2", 1, "retry budget?"));
2569        write(&a, "lessons.jsonl", std::slice::from_ref(&l1));
2570        write(&b, "lessons.jsonl", std::slice::from_ref(&l2));
2571        let edit = json!({"pb": "p1", "edit": "e1", "step": "s1", "text": "run the parser tests", "ms": 1}).to_string();
2572        write(&a, PLAYBOOKS, std::slice::from_ref(&edit));
2573        origin::append(&a, &[Origin::new(&alice_id, &lines::hash(&edit), 1)]).unwrap();
2574        let now = borg::learn::now_ms();
2575        for (h, p, who) in [(&a, &pa, "alice"), (&b, &pb, "bob")] {
2576            let ev = json!({"session_id": format!("{who}-s1"), "cwd": p.dir, "prompt": format!("{who} works")});
2577            borg::presence::write(h, &borg::presence::update(None, "work", "claude", "prompt", &ev, now, |_| (format!("{who}@x"), "repo".into()))).unwrap();
2578        }
2579        let tell = message(&b, &bob_id.public(), ALL_TO, now, "the new node's note");
2580        let old = Node::open(&pa, Some(LOCAL.parse().unwrap()), true, |x| x.alpns(vec![LEGACY_ALPN.to_vec()])).await.unwrap();
2581        let new = Node::bind(&pb, LOCAL.parse().unwrap()).await.unwrap();
2582        let (old_at, new_at) = (old.ticket(), new.ticket());
2583        let mut goal = OldGoal::new(&alice_id);
2584
2585        // The new node dials: every hive line comes once and is dropped.
2586        let (old_got, new_got) = tokio::join!(old_serve(&old, &goal), new.sync(&old_at));
2587        let (new_got, (gave, presence)) = (new_got.unwrap(), old_got.unwrap());
2588        assert_eq!((gave, presence, new_got.presence), (LEGACY_GOAL_LOGS, 1, 1));
2589        assert!(holds(&a, "lessons.jsonl", &l2) && holds(&b, "lessons.jsonl", &l1), "lessons both ways");
2590
2591        // The old node dials: its directive (bob may steer) is ignored, and the listener's echo moves
2592        // no hive line.
2593        let (old_got, new_got) = tokio::join!(old_dial(&old, &new_at, &goal), new.accept());
2594        let (new_got, (gave, _)) = (new_got.unwrap(), old_got.unwrap());
2595        assert_eq!((gave, new_got.steered, new_got.dropped), (0, 0, 0));
2596        assert!(!borg::steer::active(&b), "a 0.3.0 directive counts nowhere");
2597
2598        // The new node dials again: only the line added since comes.
2599        goal.add(&alice_id, 5, "{\"bundle\":\"new\"}");
2600        let (old_got, new_got) = tokio::join!(old_serve(&old, &goal), new.sync(&old_at));
2601        new_got.unwrap();
2602        assert_eq!(old_got.unwrap().0, 1, "what was given before is not sent again");
2603        assert!(!b.join("hive").exists(), "no hive file on the new node");
2604        assert!(!b.join(PLAYBOOKS).exists(), "the old node's playbook edit is followed, not kept");
2605        assert!(holds(&b, borg::msg::FILE, &tell) && !a.join(borg::msg::FILE).exists(), "messages stay off a 0.3.0 session");
2606        old.close().await;
2607        new.close().await;
2608    }
2609
2610    /// A revokes C; B, which has not heard yet, still takes a lesson C wrote after its revocation.
2611    /// B's next sync with A brings the new membership first: the late lesson is quarantined before
2612    /// any data moves and never reaches A, C's earlier lesson stays, and C is refused by B.
2613    #[tokio::test(flavor = "multi_thread")]
2614    async fn a_revocation_spreads_and_the_revoked_node_is_refused() {
2615        let p = crew("revoke", &["alice", "bob", "carol"]);
2616        let (a, b, c) = (home(&p[0]), home(&p[1]), home(&p[2]));
2617        let alice = Node::bind(&p[0], LOCAL.parse().unwrap()).await.unwrap();
2618        let bob = Node::bind(&p[1], LOCAL.parse().unwrap()).await.unwrap();
2619        let carol = Node::bind(&p[2], LOCAL.parse().unwrap()).await.unwrap();
2620        let (ta, tb, tc) = (alice.ticket(), bob.ticket(), carol.ticket());
2621
2622        let early = lesson("Lc1", 1, "carol's useful lesson?");
2623        write(&c, "lessons.jsonl", std::slice::from_ref(&early));
2624        let (s, r) = tokio::join!(bob.accept(), carol.sync(&tb));
2625        assert_eq!((s.unwrap().received, r.unwrap().sent), (1, 1));
2626        tokio::time::sleep(std::time::Duration::from_millis(5)).await;
2627
2628        let mut m = Membership::load(&a).unwrap().unwrap();
2629        m.revoke(&Identity::load_or_create(&a).unwrap(), &carol.public(), borg::learn::now_ms()).unwrap();
2630        m.save(&a).unwrap();
2631        tokio::time::sleep(std::time::Duration::from_millis(5)).await;
2632
2633        let late = lesson("Lc2", 1, "carol's lesson after her revocation?");
2634        append_lines(&c, "lessons.jsonl", std::slice::from_ref(&late));
2635        let (s, r) = tokio::join!(bob.accept(), carol.sync(&tb));
2636        assert_eq!((s.unwrap().received, r.unwrap().sent), (1, 1), "bob has not heard of the revocation yet");
2637        assert!(holds(&b, "lessons.jsonl", &late));
2638
2639        let (served, synced) = tokio::join!(alice.accept(), bob.sync(&ta));
2640        let (served, synced) = (served.unwrap(), synced.unwrap());
2641        assert_eq!(Membership::load(&b).unwrap().unwrap().version, m.version, "bob learned the revocation from alice");
2642        assert_eq!(synced.quarantined, 1, "carol's late lesson is set aside before any data moves");
2643        assert!(!holds(&b, "lessons.jsonl", &late) && holds(&b.join("mesh"), "quarantine.jsonl", &late));
2644        assert_eq!(served.received, 1, "alice gets carol's earlier lesson only");
2645        assert!(holds(&a, "lessons.jsonl", &early) && !holds(&a, "lessons.jsonl", &late));
2646
2647        let (served, synced) = tokio::join!(bob.accept(), carol.sync(&tb));
2648        assert!(served.unwrap_err().contains("not a member"), "bob turns carol away");
2649        assert!(synced.is_err());
2650        let err = bob.sync(&tc).await.unwrap_err();
2651        assert!(err.contains("not a member") && err.contains("refusing to connect"), "{err}");
2652        for n in [alice, bob, carol] {
2653            n.close().await;
2654        }
2655    }
2656
2657    /// After days apart, two nodes holding 2000 common lessons and 5 new ones between them move
2658    /// exactly the 5 lines and a few dozen hashes, not the history.
2659    #[tokio::test(flavor = "multi_thread")]
2660    async fn catch_up_transfers_only_the_difference() {
2661        let (pa, pb) = team("catch-up");
2662        let (a, b) = (home(&pa), home(&pb));
2663        let common: Vec<String> = (0..2000).map(|i| lesson(&format!("L{i}"), 1, &format!("question {i}?"))).collect();
2664        write(&a, "lessons.jsonl", &common);
2665        write(&b, "lessons.jsonl", &common);
2666        append_lines(&a, "lessons.jsonl", &(0..3).map(|i| lesson(&format!("A{i}"), 1, "alice offline?")).collect::<Vec<_>>());
2667        append_lines(&b, "lessons.jsonl", &(0..2).map(|i| lesson(&format!("B{i}"), 1, "bob offline?")).collect::<Vec<_>>());
2668        let alice = Node::bind(&pa, LOCAL.parse().unwrap()).await.unwrap();
2669        let bob = Node::bind(&pb, LOCAL.parse().unwrap()).await.unwrap();
2670        let ticket = alice.ticket();
2671        let (served, synced) = tokio::join!(alice.accept(), bob.sync(&ticket));
2672        let (served, synced) = (served.unwrap(), synced.unwrap());
2673        assert_eq!((synced.received, synced.sent, served.received, served.sent), (3, 2, 2, 3));
2674        let hashes = synced.hashes + served.hashes;
2675        let fps = synced.fps + served.fps;
2676        eprintln!("catch-up: 5 of 2005 lines moved, {hashes} hashes and {fps} fingerprints in {} rounds", synced.rounds);
2677        assert!(hashes <= 2 * crate::recon::LEAF * 5, "hashes ∝ difference, not history: {hashes}");
2678        assert!(fps <= 2 * (1 + 16 * 5 * synced.rounds), "{fps}");
2679        assert_eq!(lines::read(&a.join("lessons.jsonl")), lines::read(&b.join("lessons.jsonl")));
2680        alice.close().await;
2681        bob.close().await;
2682    }
2683
2684    /// A reasoning excerpt goes from its author to exactly its 2 rendezvous replicas, and no
2685    /// member accepts another member's excerpt passed off as a replica.
2686    #[tokio::test(flavor = "multi_thread")]
2687    async fn reasoning_goes_to_two_replicas() {
2688        let p = crew("replicas", &["alice", "bob", "carol", "dave"]);
2689        let a = home(&p[0]);
2690        let excerpt = json!({"commit": "abc", "task": "fix the lock", "proposal": "take it in order"}).to_string();
2691        write(&a, REASONING, std::slice::from_ref(&excerpt));
2692        let alice = Node::bind(&p[0], LOCAL.parse().unwrap()).await.unwrap();
2693        let m = Membership::load(&a).unwrap().unwrap();
2694        let chosen = replicas(&lines::hash(&excerpt), &m, &alice.public());
2695        assert_eq!(chosen.len(), 2);
2696        let mut holders = Vec::new();
2697        for q in &p[1..] {
2698            let n = Node::bind(q, LOCAL.parse().unwrap()).await.unwrap();
2699            let ticket = alice.ticket();
2700            let (served, synced) = tokio::join!(alice.accept(), n.sync(&ticket));
2701            served.unwrap();
2702            synced.unwrap();
2703            if holds(&home(q), REASONING, &excerpt) {
2704                holders.push(n.public());
2705            }
2706            n.close().await;
2707        }
2708        holders.sort();
2709        let mut want = chosen.clone();
2710        want.sort();
2711        assert_eq!(holders, want, "exactly the two replicas hold it");
2712        alice.close().await;
2713    }
2714
2715    /// With the default bind (0.0.0.0) no ticket says 0.0.0.0, and the listener remembers the
2716    /// dialler at the IP it saw it come from, with the port it announced: it can dial back.
2717    #[tokio::test(flavor = "multi_thread")]
2718    async fn a_node_bound_to_any_address_is_dialled_back_where_it_was_seen() {
2719        let (pa, pb) = team("any-addr");
2720        let alice = Node::bind(&pa, "0.0.0.0:0".parse().unwrap()).await.unwrap();
2721        let bob = Node::bind(&pb, "0.0.0.0:0".parse().unwrap()).await.unwrap();
2722        let (pa_port, pb_port) = (alice.addr().unwrap().port(), bob.addr().unwrap().port());
2723        for t in alice.tickets() {
2724            assert!(!t.contains("@0.0.0.0:") && t.ends_with(&format!(":{pa_port}")), "{t}");
2725        }
2726        let ticket = format!("{}@127.0.0.1:{pa_port}", alice.public());
2727        let (served, synced) = tokio::join!(alice.accept(), bob.sync(&ticket));
2728        served.unwrap();
2729        synced.unwrap();
2730        assert_eq!(peers::load(&home(&pa)).get(&bob.public()), Some(&format!("127.0.0.1:{pb_port}")), "bob is remembered where alice saw him");
2731        // And alice can dial bob back there.
2732        let known = peers::load(&home(&pa));
2733        let back = format!("{}@{}", bob.public(), known[&bob.public()]);
2734        let (served, synced) = tokio::join!(bob.accept(), alice.sync(&back));
2735        served.unwrap();
2736        synced.unwrap();
2737        // A one-shot dialler announces nothing: its port is gone a moment later.
2738        let quiet = Node::bind(&pb, "0.0.0.0:0".parse().unwrap()).await.unwrap().quiet();
2739        assert_eq!(quiet.listen(), None);
2740        assert_eq!(reachable("0.0.0.0:7878", Some("10.0.0.5:51000".parse().unwrap())), Some("10.0.0.5:7878".parse().unwrap()));
2741        assert_eq!(reachable("0.0.0.0:7878", None), None);
2742        for n in [alice, bob, quiet] {
2743            n.close().await;
2744        }
2745    }
2746
2747    /// B already holds a lesson C wrote after its revocation (from C, before B knew). The admin
2748    /// then hands B the new membership with `join`: the lesson is quarantined right there, not at
2749    /// some later sync (which, B holding the newest version already, would find nothing newer).
2750    /// The same for a node whose membership file another process replaced: `enforce` catches it.
2751    #[tokio::test(flavor = "multi_thread")]
2752    async fn join_and_enforce_quarantine_what_the_new_membership_refuses() {
2753        let p = crew("join-q", &["alice", "bob", "carol", "dave"]);
2754        let (a, b, c, d) = (home(&p[0]), home(&p[1]), home(&p[2]), home(&p[3]));
2755        let bob = Node::bind(&p[1], LOCAL.parse().unwrap()).await.unwrap();
2756        let carol = Node::bind(&p[2], LOCAL.parse().unwrap()).await.unwrap();
2757        let dave = Node::bind(&p[3], LOCAL.parse().unwrap()).await.unwrap();
2758        let early = lesson("Lc1", 1, "carol's useful lesson?");
2759        write(&c, "lessons.jsonl", std::slice::from_ref(&early));
2760        carol.stamp().unwrap();
2761        tokio::time::sleep(std::time::Duration::from_millis(5)).await;
2762        let (m, _) = origin::revoke(&a, &Identity::load_or_create(&a).unwrap(), &carol.public(), borg::learn::now_ms()).unwrap();
2763        tokio::time::sleep(std::time::Duration::from_millis(5)).await;
2764        let late = lesson("Lc2", 1, "carol's lesson after her revocation?");
2765        append_lines(&c, "lessons.jsonl", std::slice::from_ref(&late));
2766        for (node, h) in [(&bob, &b), (&dave, &d)] {
2767            let t = node.ticket();
2768            let (s, r) = tokio::join!(node.accept(), carol.sync(&t));
2769            s.unwrap();
2770            r.unwrap();
2771            assert!(holds(h, "lessons.jsonl", &late) && holds(h, "lessons.jsonl", &early));
2772        }
2773
2774        let handed = std::fs::read_to_string(crate::membership::path(&a)).unwrap();
2775        let (got, moved) = origin::join(&b, "work", &crate::membership::parse_docs(&handed).unwrap()).unwrap();
2776        assert_eq!((got.version, moved), (m.version, 1));
2777        assert!(!holds(&b, "lessons.jsonl", &late) && holds(&b, "lessons.jsonl", &early), "quarantined at join");
2778        assert!(origin::join(&b, "work", &crate::membership::parse_docs(&handed).unwrap()).is_err(), "nothing newer the second time");
2779
2780        m.save(&d).unwrap(); // installed behind the daemon's back
2781        assert_eq!(dave.enforce().unwrap(), 1);
2782        assert_eq!(dave.enforce().unwrap(), 0, "once per membership");
2783        assert!(!holds(&d, "lessons.jsonl", &late));
2784        for n in [bob, carol, dave] {
2785            n.close().await;
2786        }
2787    }
2788
2789    fn git(repo: &Path, args: &[&str]) -> String {
2790        let o = std::process::Command::new("git")
2791            .arg("-C")
2792            .arg(repo)
2793            .args(args)
2794            .env("GIT_CONFIG_GLOBAL", "/dev/null")
2795            .env("GIT_CONFIG_NOSYSTEM", "1")
2796            .output()
2797            .unwrap();
2798        assert!(o.status.success(), "git {args:?}: {}", String::from_utf8_lossy(&o.stderr));
2799        String::from_utf8_lossy(&o.stdout).trim().to_string()
2800    }
2801
2802    /// The author hands an excerpt to its two replicas and goes offline; a member that is no
2803    /// replica asks one of them by commit and gets it back as a git note.
2804    #[tokio::test(flavor = "multi_thread")]
2805    async fn reasoning_is_fetched_from_a_replica_while_its_author_is_offline() {
2806        let p = crew("fetch", &["alice", "bob", "carol", "dave"]);
2807        let a = home(&p[0]);
2808        let repo = p[3].base.join("work");
2809        std::fs::create_dir_all(&repo).unwrap();
2810        git(&repo, &["init", "-q"]);
2811        git(&repo, &["-c", "user.email=ann@example.com", "-c", "user.name=Ann", "commit", "-q", "--allow-empty", "-m", "take the lock first"]);
2812        let commit = git(&repo, &["rev-parse", "HEAD"]);
2813        let excerpt = json!({"kind": "reasoning", "commit": commit, "repo": "work", "session": "s-9", "task": "fix the race",
2814            "proposal": "take the lock before reading the flag", "episodes": [], "ms": 1})
2815        .to_string();
2816        write(&a, REASONING, std::slice::from_ref(&excerpt));
2817        let alice = Node::bind(&p[0], LOCAL.parse().unwrap()).await.unwrap();
2818        let chosen = replicas(&lines::hash(&excerpt), &Membership::load(&a).unwrap().unwrap(), &alice.public());
2819        let mut others = Vec::new();
2820        for q in &p[1..] {
2821            let n = Node::bind(q, LOCAL.parse().unwrap()).await.unwrap();
2822            let t = alice.ticket();
2823            let (served, synced) = tokio::join!(alice.accept(), n.sync(&t));
2824            served.unwrap();
2825            synced.unwrap();
2826            others.push((q.clone(), n));
2827        }
2828        alice.close().await; // the author is offline from here on
2829        let (holders, askers): (Vec<_>, Vec<_>) = others.into_iter().partition(|(_, n)| chosen.contains(&n.public()));
2830        let (asker_place, asker) = askers.into_iter().next().expect("one member is no replica");
2831        let holder = &holders[0].1;
2832        assert!(!holds(&home(&asker_place), REASONING, &excerpt));
2833
2834        let short = commit[..10].to_string();
2835        let t = holder.ticket();
2836        let (served, fetched) = tokio::join!(holder.accept(), asker.fetch(&t, std::slice::from_ref(&short)));
2837        assert_eq!(served.unwrap().answered, 1, "a replica answers for an excerpt it did not write");
2838        assert_eq!(fetched.unwrap().received, 1);
2839        let written = borg::notes::restore(&repo, &home(&asker_place), std::slice::from_ref(&commit)).unwrap();
2840        assert_eq!(written, vec![commit.clone()]);
2841        assert!(git(&repo, &["notes", "--ref=refs/notes/borg", "show", &commit]).contains("take the lock before reading the flag"));
2842
2843        // Nobody is handed an excerpt it did not ask for.
2844        let (served, synced) = tokio::join!(holder.accept(), asker.sync(&t));
2845        assert_eq!((served.unwrap().answered, synced.unwrap().received), (0, 0));
2846        asker.close().await;
2847        for (_, n) in holders {
2848            n.close().await;
2849        }
2850    }
2851
2852    /// A steer from the node whose member key is `node` (its engineer at a terminal) to `to`.
2853    fn steering(node: &str, kind: borg::msg::Kind, to: &str, ms: i64, text: &str) -> borg::msg::Message {
2854        let from = borg::msg::Sender { by: "engineer:bob@x".into(), name: "bob@x".into(), harness: "terminal".into(), node: node.into() };
2855        borg::msg::Message::new(kind, &from, to, None, ms, text)
2856    }
2857
2858    /// Presence and steering ride the session: bob (role steer) sees alice's drone and pauses it, a
2859    /// message line whose origin is bob's node; alice admits it under bob's role and her hook refuses
2860    /// tools until bob's resume, sent at once after. An observer's pause is stored but refused and
2861    /// logged; once bob is revoked, his pause no longer holds.
2862    #[tokio::test(flavor = "multi_thread")]
2863    async fn presence_and_steering_travel_between_members() {
2864        use borg::msg::Kind;
2865        let (pa, pb) = team("live");
2866        let (a, b) = (home(&pa), home(&pb));
2867        let (alice_id, bob_id) = (Identity::load_or_create(&a).unwrap(), Identity::load_or_create(&b).unwrap());
2868        let (alice_key, bob_key) = (alice_id.public(), bob_id.public());
2869        let mut m = Membership::load(&a).unwrap().unwrap();
2870        m.invite(&alice_id, &bob_key, "steer").unwrap();
2871        m.save(&a).unwrap(); // every version goes into the history, or the chain breaks
2872        let observer = Identity::load_or_create(&a.join("observer")).unwrap();
2873        m.invite(&alice_id, &observer.public(), "observe").unwrap();
2874        m.save(&a).unwrap(); // bob adopts both versions at the first sync
2875        let now = borg::learn::now_ms();
2876        let ev = json!({"session_id": "alice-s1", "cwd": pa.dir, "prompt": "refactor the recon loop"});
2877        borg::presence::write(&a, &borg::presence::update(None, "work", "claude", "prompt", &ev, now, |_| ("alice@x".into(), "repo".into()))).unwrap();
2878
2879        let alice = Node::bind(&pa, LOCAL.parse().unwrap()).await.unwrap();
2880        let bob = Node::bind(&pb, LOCAL.parse().unwrap()).await.unwrap();
2881        let ticket = alice.ticket();
2882        let (served, synced) = tokio::join!(alice.accept(), bob.sync(&ticket));
2883        served.unwrap();
2884        assert_eq!(synced.unwrap().presence, 1);
2885        let ds = borg::top::drones(&b, true, now);
2886        assert_eq!((ds.len(), ds[0].node.as_str(), ds[0].p.prompt.as_str()), (1, alice_key.as_str(), "refactor the recon loop"));
2887        let lessons = std::fs::read_to_string(b.join("lessons.jsonl")).unwrap_or_default();
2888        assert!(!lessons.contains("recon loop"), "never in lessons");
2889
2890        let to = format!("alice-s1@{alice_key}");
2891        let pre = json!({"session_id": "alice-s1", "cwd": pa.dir, "tool_name": "Bash"});
2892        let refused = |at: i64| borg::steer::around(&a, "pre_tool", &pre, Some(now), at, || None);
2893        let round = || async {
2894            let (served, synced) = tokio::join!(alice.accept(), bob.sync(&ticket));
2895            synced.unwrap();
2896            served.unwrap()
2897        };
2898        borg::msg::send(&b, steering(&bob_key, Kind::Pause, &to, now, ""), now).unwrap();
2899        let served = round().await;
2900        assert_eq!((served.steered, served.dropped), (1, 0));
2901        let deny = refused(now + 1).expect("paused");
2902        let why = deny["hookSpecificOutput"]["permissionDecisionReason"].as_str().unwrap();
2903        assert!(why.contains(&format!("paused by bob@x (node {}, role steer)", &bob_key[..12])), "{why}");
2904
2905        // An observer's pause, handed over signed by its node: stored, refused, logged.
2906        let by_observer = serde_json::to_string(&steering(&observer.public(), Kind::Pause, ALL_TO, now, "")).unwrap();
2907        let h = lines::hash(&by_observer);
2908        let mut report = Report::default();
2909        let got = vec![Signed { origin: Origin::new(&observer, &h, now), line: by_observer.clone() }];
2910        alice.store_lines(&a.join(borg::msg::FILE), got, &BTreeSet::from([h]), &members(&a).unwrap(), None, &mut report).unwrap();
2911        assert_eq!((report.received, report.steered, report.dropped), (1, 0, 1));
2912        let log = std::fs::read_to_string(borg::steer::dropped_path(&a)).unwrap();
2913        assert!(log.contains("role observe may not steer"), "{log}");
2914
2915        // Bob's resume, a moment after his pause: not paced away, and it lifts the pause.
2916        borg::msg::send(&b, steering(&bob_key, Kind::Resume, &to, now + 2, ""), now + 2).unwrap();
2917        assert_eq!(round().await.steered, 1);
2918        assert_eq!(refused(now + 3), None);
2919
2920        // Bob pauses again, then is revoked: his pause stops holding at alice.
2921        borg::msg::send(&b, steering(&bob_key, Kind::Pause, &to, now + 4, ""), now + 4).unwrap();
2922        assert_eq!(round().await.steered, 1);
2923        assert!(refused(now + 5).is_some(), "paused again");
2924        let mut m = Membership::load(&a).unwrap().unwrap();
2925        m.revoke(&alice_id, &bob_key, now + 5).unwrap();
2926        m.save(&a).unwrap();
2927        alice.enforce().unwrap();
2928        assert!(borg::steer::read_admitted(&a).iter().all(|x| x.node != bob_key), "the revoked node's admissions are gone");
2929        assert_eq!(refused(now + 6), None, "and the pause no longer holds");
2930        alice.close().await;
2931        bob.close().await;
2932    }
2933
2934    /// `borg no` on bob's node for alice's drone: a veto message, admitted and recorded once on alice's
2935    /// node (credit, episode, stat), shown to her drone once, and back on bob's node as a team outcome,
2936    /// so both nodes' beliefs count the veto once.
2937    #[tokio::test(flavor = "multi_thread")]
2938    async fn a_veto_on_one_node_reaches_the_drones_node_once() {
2939        let (pa, pb) = team("veto");
2940        let (a, b) = (home(&pa), home(&pb));
2941        let (alice_id, bob_id) = (Identity::load_or_create(&a).unwrap(), Identity::load_or_create(&b).unwrap());
2942        let mut m = Membership::load(&a).unwrap().unwrap();
2943        m.invite(&alice_id, &bob_id.public(), "steer").unwrap();
2944        m.save(&a).unwrap();
2945        let now = borg::learn::now_ms();
2946        let ev = json!({"session_id": "alice-s1", "cwd": pa.dir, "prompt": "look merges up"});
2947        borg::presence::write(&a, &borg::presence::update(None, "work", "claude", "prompt", &ev, now, |_| ("alice@x".into(), "repo".into()))).unwrap();
2948        write(&a, "lessons.jsonl", &[lesson("L1", 1, "how are merges found?")]);
2949        let asked = json!({"kind": "asked", "session": "alice-s1", "lessons": ["L1"], "ms": now - 10, "repo": "repo"}).to_string();
2950        write(&a, "outcomes.jsonl", &[asked]);
2951        let alice = Node::bind(&pa, LOCAL.parse().unwrap()).await.unwrap();
2952        let bob = Node::bind(&pb, LOCAL.parse().unwrap()).await.unwrap();
2953        let ticket = alice.ticket();
2954        let (served, synced) = tokio::join!(alice.accept(), bob.sync(&ticket));
2955        let _ = (served.unwrap(), synced.unwrap());
2956
2957        let to = format!("alice-s1@{}", alice_id.public());
2958        let veto = steering(&bob_id.public(), borg::msg::Kind::Veto, &to, now, "never look merges up in the commit log");
2959        borg::msg::send(&b, veto, now).unwrap();
2960        assert!(!b.join("episodes.jsonl").exists(), "the issuing node records no veto of its own");
2961        for _ in 0..2 {
2962            let (served, synced) = tokio::join!(alice.accept(), bob.sync(&ticket));
2963            let _ = (served.unwrap(), synced.unwrap());
2964        }
2965        let vetoed =
2966            |h: &Path| borg::outcome::read(&h.join("outcomes.jsonl")).into_iter().filter(|o| o.kind == borg::outcome::VETOED && o.lessons == ["L1"]).count();
2967        assert_eq!(vetoed(&a), 1, "credited once on the drone's node, over two rounds");
2968        let eps = std::fs::read_to_string(a.join("episodes.jsonl")).unwrap();
2969        assert_eq!(eps.lines().count(), 1);
2970        assert!(eps.contains("\"kind\":\"veto\"") && eps.contains("bob@x") && eps.contains("never look merges up"), "{eps}");
2971        let stats: Vec<_> = borg::stats::events(&a).into_iter().filter(|e| e.kind == borg::stats::VETO).collect();
2972        assert_eq!((stats.len(), stats[0].why.as_str()), (1, "mesh"));
2973        // Alice's drone is told once, labelled with bob's node and role.
2974        let caps = borg::harness::by_name("claude").unwrap().caps();
2975        let prompt = json!({"session_id": "alice-s1", "cwd": pa.dir});
2976        let out = borg::msg::deliver(&a, caps, "prompt", &prompt, Some(now - 10), now + 1, None).expect("shown");
2977        let ctx = out["hookSpecificOutput"]["additionalContext"].as_str().unwrap();
2978        assert!(ctx.contains(&format!("[borg] Veto from bob@x (node {}, role steer)", &bob_id.public()[..12])), "{ctx}");
2979        assert_eq!(borg::msg::deliver(&a, caps, "prompt", &prompt, Some(now - 10), now + 2, None), None, "once");
2980        // Both nodes' beliefs count it once: alice's from her own record, bob's from the team outcome.
2981        let l1 = [borg::conscience::Challenge { id: "L1".into(), ..Default::default() }];
2982        let belief = |h: &Path| borg::memory::derive(&borg::outcome::read_team(&h.join("lessons.jsonl")), &l1, None)["L1"].vetoed;
2983        assert_eq!((belief(&a), belief(&b)), (1, 1));
2984        assert!(!b.join("outcomes.jsonl").exists() && !b.join("episodes.jsonl").exists(), "bob's node holds only the team copy");
2985        alice.close().await;
2986        bob.close().await;
2987    }
2988
2989    #[tokio::test(flavor = "multi_thread")]
2990    async fn a_local_collectives_presence_and_steering_never_leave() {
2991        let p = node("live-private", "local");
2992        let h = home(&p);
2993        let now = borg::learn::now_ms();
2994        let ev = json!({"session_id": "s1", "cwd": p.dir, "prompt": "private diary"});
2995        borg::presence::write(&h, &borg::presence::update(None, "work", "claude", "prompt", &ev, now, |_| ("me@x".into(), "r".into()))).unwrap();
2996        let me = Identity::load_or_create(&h).unwrap().public();
2997        borg::msg::send(&h, steering(&me, borg::msg::Kind::Pause, ALL_TO, now, "lunch"), now).unwrap();
2998        let held = borg::steer::read_admitted(&h);
2999        assert_eq!((held.len(), held[0].role), (1, borg::steer::Role::Local), "this machine's engineer steers it");
3000        assert!(Node::bind(&p, LOCAL.parse().unwrap()).await.is_err(), "no socket at all");
3001    }
3002
3003    /// A pause and its resume from one sender to one session both travel at once (send does not pace a
3004    /// steer). Of more unread tells from one sender to one recipient within `V1_RATE_MS` than any window
3005    /// lets through (`WINDOW_CAP`), the oldest are outside; one acknowledged is in, and so is one a rate
3006    /// slot later.
3007    #[test]
3008    fn the_window_bounds_a_burst_and_never_a_steer() {
3009        use borg::msg::{Kind, Seen, V1_RATE_MS, WINDOW_CAP};
3010        let (node, recipient) = ("a".repeat(64), "b".repeat(64));
3011        let to = format!("s1@{recipient}");
3012        let steers = [steering(&node, Kind::Pause, &to, 1000, ""), steering(&node, Kind::Resume, &to, 1001, "")];
3013        let burst: Vec<borg::msg::Message> = (0..WINDOW_CAP + 2).map(|i| steering(&node, Kind::Tell, &to, 2000 + i as i64, &format!("t{i}"))).collect();
3014        let later = steering(&node, Kind::Tell, &to, 2000 + burst.len() as i64 + V1_RATE_MS, "a slot later");
3015        let line = |m: &borg::msg::Message| serde_json::to_string(m).unwrap();
3016        let mut set: BTreeMap<Hash, String> = steers.iter().chain(&burst).chain([&later]).map(line).map(|l| (lines::hash(&l), l)).collect();
3017        let within = |set: &BTreeMap<Hash, String>| {
3018            let w = window(set);
3019            steers.iter().chain(&burst).chain([&later]).map(|m| w.contains(&lines::hash(&line(m)))).collect::<Vec<bool>>()
3020        };
3021        let mut want = vec![true; steers.len() + burst.len() + 1];
3022        (want[2], want[3]) = (false, false);
3023        assert_eq!(within(&set), want, "steers in; of {} unread within a rate slot, the newest {WINDOW_CAP}", burst.len());
3024        let seen = serde_json::to_string(&Seen::new("s1", &recipient, 3000, &[&burst[0].id])).unwrap();
3025        set.insert(lines::hash(&seen), seen);
3026        want[2] = true;
3027        assert_eq!(within(&set), want, "the first acknowledged: in, and the second still one too many");
3028    }
3029
3030    /// What a `borg/mesh/13` peer is offered: a pause and then `per_node + 1` newer tells from its node
3031    /// keep the pause (steers have their own share, as 0.5.0 kept them), the oldest tell goes, and no v2
3032    /// text or seen line is offered at all.
3033    #[test]
3034    fn the_v1_window_keeps_a_pause_and_offers_only_v1_lines() {
3035        use borg::msg::Kind;
3036        let (node, per) = ("a".repeat(64), 5);
3037        let to = |i: usize| format!("s{i}@{}", "b".repeat(64));
3038        let ms: Vec<borg::msg::Message> = std::iter::once(steering(&node, Kind::Pause, &to(0), 1000, ""))
3039            .chain((0..=per).map(|i| steering(&node, Kind::Tell, &to(i), 1001 + i as i64, "hi")))
3040            .collect();
3041        let long = steering(&node, Kind::Tell, &to(9), 2000, &"w".repeat(borg::msg::V1_TEXT + 1));
3042        let seen = seen_of(&node, "s1", 2001, &[&ms[1].id]);
3043        let mut set: BTreeMap<Hash, String> = ms.iter().chain([&long]).map(|m| serde_json::to_string(m).unwrap()).map(|l| (lines::hash(&l), l)).collect();
3044        set.insert(lines::hash(&seen), seen.clone());
3045        let w = v1_window(&set, per);
3046        let kept = |m: &borg::msg::Message| w.contains(&lines::hash(&serde_json::to_string(m).unwrap()));
3047        assert!(kept(&ms[0]), "the pause travels");
3048        assert_eq!((kept(&ms[1]), ms[2..].iter().filter(|m| kept(m)).count()), (false, per), "the oldest tell went");
3049        assert!(!kept(&long) && !w.contains(&lines::hash(&seen)), "a v2 text and a seen line are not 0.5.0's to read");
3050    }
3051
3052    /// What `send` writes is within the mesh's window, on every node and however much of the log the TTL
3053    /// and the bound have pruned since: a log built by `send` and `ack` alone (sessions of this node and
3054    /// of another, a broadcast and the engineer; some acknowledged, some answered, some refused, over
3055    /// three TTLs), each tell in it within `window`, also over every later stretch of it, as a node
3056    /// holding only that stretch sees it. Not vacuous: over those stretches `window_for` admits less than
3057    /// `send` did.
3058    #[test]
3059    fn what_send_wrote_stays_within_the_window_however_the_log_is_pruned() {
3060        use borg::msg::{Address, Kind, Message, SendError, Sender};
3061        let home = std::env::temp_dir().join(format!("borg-mesh-sent-{}", std::process::id()));
3062        let _ = std::fs::remove_dir_all(&home);
3063        std::fs::create_dir_all(&home).unwrap();
3064        let here = borg::msg::node(&home);
3065        let senders = ["s1", "s2", "engineer:e@x"];
3066        let recipients = [format!("r1@{here}"), format!("r1@{here}"), format!("r2@{here}"), format!("r3@{}", "d".repeat(64)), "all".into(), "engineer".into()];
3067        let mut seed = 11u64;
3068        let mut pick = |n: usize| {
3069            seed = seed.wrapping_mul(6364136223846793005).wrapping_add(1442695040888963407);
3070            (seed >> 33) as usize % n
3071        };
3072        let start = 10 * borg::msg::TTL_MS;
3073        let (mut ms, mut sent, mut acks, mut refused) = (start, Vec::<Message>::new(), 0, 0);
3074        while ms < start + 3 * borg::msg::TTL_MS {
3075            ms += [1_000, 1_000, borg::msg::V1_RATE_MS, 3_600_000, 6 * 3_600_000][pick(5)];
3076            let from = Sender { by: senders[pick(senders.len())].into(), name: "One of One".into(), harness: "claude".into(), node: here.clone() };
3077            // One of the latest, so acknowledgements and answers open windows still in use.
3078            let earlier = (!sent.is_empty()).then(|| sent[sent.len() - 1 - pick(sent.len().min(4))].clone());
3079            // Its recipient here shown it (a broadcast: any other session here), or answering it.
3080            let recipient = earlier.as_ref().and_then(|m| match Address::parse(&m.to) {
3081                Some(Address::Session { id, node: Some(n) }) if n == here => Some(id),
3082                Some(Address::All) => Some("r9".to_string()),
3083                _ => None,
3084            });
3085            match (pick(6), earlier, recipient) {
3086                (0 | 1, Some(m), Some(who)) => {
3087                    borg::msg::ack(&home, &who, &[&m.id], ms).unwrap();
3088                    acks += 1;
3089                }
3090                (2, Some(m), Some(who)) if !borg::msg::is_engineer(&m.by) => {
3091                    let from = Sender { by: who, name: "Two of Two".into(), harness: "omp".into(), node: here.clone() };
3092                    let _ = borg::msg::send(&home, Message::new(Kind::Tell, &from, &format!("{}@{here}", m.by), Some(&m.id), ms, "re"), ms);
3093                }
3094                _ => match borg::msg::send(&home, Message::new(Kind::Tell, &from, &recipients[pick(recipients.len())], None, ms, "hi"), ms) {
3095                    Ok(m) => sent.push(m),
3096                    Err(SendError::Window { .. }) => refused += 1,
3097                    Err(e) => panic!("{e}"),
3098                },
3099            }
3100        }
3101        let held: Vec<String> = std::fs::read_to_string(borg::msg::path(&home)).unwrap().lines().map(String::from).collect();
3102        let talk = |l: &str| Line::parse(l).and_then(|x| x.message().cloned()).filter(|m| !m.kind.privileged());
3103        assert!(acks > 10 && refused > 10 && held.iter().filter(|l| talk(l).is_some()).count() < sent.len(), "{acks} acks, {refused} refused, pruned");
3104        let mut fewer = 0;
3105        for cut in (0..held.len()).step_by(5) {
3106            let set: BTreeMap<Hash, String> = held[cut..].iter().map(|l| (lines::hash(l), l.clone())).collect();
3107            let (w, parsed) = (window(&set), held[cut..].iter().filter_map(|l| Line::parse(l)).collect::<Vec<Line>>());
3108            for (l, m) in held[cut..].iter().filter_map(|l| talk(l).map(|m| (l, m))) {
3109                assert!(w.contains(&lines::hash(l)), "from line {cut} on: {l}");
3110                fewer += usize::from(!borg::msg::window_for(&parsed, m.author(), &m.to, ms).admits(&m.id));
3111            }
3112        }
3113        assert!(fewer > 0, "some stretch's replay admits less than send did");
3114        let _ = std::fs::remove_dir_all(&home);
3115    }
3116
3117    /// `Live` still carries the `directives` a 0.3.0 peer requires, always empty, and takes whatever
3118    /// such a peer puts there without reading it.
3119    #[test]
3120    fn live_keeps_an_empty_directives_field_for_0_3_0() {
3121        let out = serde_json::to_value(Msg::Live { presence: Vec::new(), directives: Directives }).unwrap();
3122        assert_eq!(out, json!({"t": "live", "presence": [], "directives": []}));
3123        let theirs = json!({"t": "live", "presence": [], "directives": [{"kind": "pause", "sig": "00"}], "beats": []});
3124        assert!(matches!(serde_json::from_value::<Msg>(theirs).unwrap(), Msg::Live { .. }));
3125        assert!(matches!(serde_json::from_value::<Msg>(json!({"t": "live", "presence": []})).unwrap(), Msg::Live { .. }));
3126    }
3127
3128    /// The admin answers every dial-in (joins and syncs) until the test ends.
3129    fn answer(n: std::sync::Arc<Node>) {
3130        tokio::spawn(async move {
3131            while let Some(i) = n.incoming().await {
3132                let n = n.clone();
3133                tokio::spawn(async move {
3134                    let _ = n.handle(i).await;
3135                });
3136            }
3137        });
3138    }
3139
3140    fn ticket(admin: &Node, role: &str, ttl_ms: i64, uses: u32) -> Ticket {
3141        let home = admin.home();
3142        let m = Membership::load(home).unwrap().unwrap();
3143        let invite = invite::issue(home, &admin.me, &m, role, ttl_ms, uses, borg::learn::now_ms()).unwrap();
3144        Ticket::decode(&Ticket { invite, relay: admin.relay.clone(), hints: admin.hints() }.encode()).unwrap()
3145    }
3146
3147    /// Bob and carol join alice's collective with tickets alone (no key exchanged by hand, no file
3148    /// handed over): each gets the signed membership with the ticket's role and alice's lesson at
3149    /// once. A spent ticket is refused for carol but welcomes bob again; an expired one is refused.
3150    #[tokio::test(flavor = "multi_thread")]
3151    async fn three_nodes_join_by_ticket_and_sync_a_lesson() {
3152        let pa = crew("ticket", &["alice"]).remove(0);
3153        let (pb, pc) = (node("ticket-bob", "team"), node("ticket-carol", "team"));
3154        let l1 = lesson("L1", 1, "who owns the lock?");
3155        write(&home(&pa), "lessons.jsonl", std::slice::from_ref(&l1));
3156        let alice = std::sync::Arc::new(Node::bind(&pa, LOCAL.parse().unwrap()).await.unwrap());
3157        answer(alice.clone());
3158
3159        let one = ticket(&alice, "member", 60_000, 1);
3160        let bob = Node::joining(&pb, LOCAL.parse().unwrap()).await.unwrap().quiet();
3161        let (m, rep) = bob.join(&one).await.unwrap();
3162        assert_eq!(m.role_of(&bob.public()), Some("member"));
3163        assert_eq!(rep.received, 1, "bob syncs at once");
3164        assert!(holds(&home(&pb), "lessons.jsonl", &l1));
3165        assert_eq!(peers::load(&home(&pb)).get(&alice.public()), Some(&addr_of(&alice)), "bob knows where alice is");
3166        assert!(peers::load(&home(&pa)).contains_key(&bob.public()), "alice knows bob, by key at least");
3167
3168        let carol = Node::joining(&pc, LOCAL.parse().unwrap()).await.unwrap().quiet();
3169        let err = carol.join(&one).await.unwrap_err();
3170        assert!(err.contains("already used"), "{err}");
3171        assert!(Membership::load(&home(&pc)).unwrap().is_none() && !home(&pc).join("lessons.jsonl").exists(), "carol got nothing");
3172        assert!(bob.join(&one).await.is_ok(), "bob again (a lost answer): welcomed");
3173
3174        let gone = ticket(&alice, "member", 1, 1);
3175        tokio::time::sleep(std::time::Duration::from_millis(5)).await;
3176        assert!(carol.join(&gone).await.unwrap_err().contains("expired"));
3177
3178        let two = ticket(&alice, "observe", 60_000, 1);
3179        let (m, rep) = carol.join(&two).await.unwrap();
3180        assert_eq!(m.role_of(&carol.public()), Some("observe"));
3181        assert_eq!(rep.received, 1);
3182        let held = Membership::load(&home(&pa)).unwrap().unwrap();
3183        assert_eq!((held.role_of(&bob.public()), held.role_of(&carol.public())), (Some("member"), Some("observe")));
3184        assert!(invite::pending(alice.home(), borg::learn::now_ms()).is_empty(), "both tickets spent");
3185
3186        // Bob learns of carol at his next sync with alice: the signed chain spreads as before.
3187        assert_eq!(Membership::load(&home(&pb)).unwrap().unwrap().role_of(&carol.public()), None);
3188        bob.sync(&format!("{}@{}", alice.public(), addr_of(&alice))).await.unwrap();
3189        assert_eq!(Membership::load(&home(&pb)).unwrap().unwrap().role_of(&carol.public()), Some("observe"));
3190        for n in [bob, carol] {
3191            n.close().await;
3192        }
3193        alice.close().await;
3194    }
3195
3196    /// The NAT case as far as one machine can show it: neither node has a direct address (no UDP
3197    /// socket at all), the ticket carries none, so the join and the sync can only go through the
3198    /// (local, self-signed) relay both use.
3199    #[tokio::test(flavor = "multi_thread")]
3200    async fn join_and_sync_go_through_the_relay_when_no_direct_address_is_known() {
3201        let (_map, url, _server) = iroh::test_utils::run_relay_server().await.unwrap();
3202        let pa = crew("relay", &["alice"]).remove(0);
3203        let pb = node("relay-bob", "team");
3204        for p in [&pa, &pb] {
3205            let cfg = json!({"default": "off", "collectives": [{"name": "work", "paths": [p.dir], "share": "team", "relay": url.to_string()}]});
3206            std::fs::write(&p.config, cfg.to_string()).unwrap();
3207        }
3208        let l1 = lesson("L1", 1, "does this cross the NAT?");
3209        write(&home(&pa), "lessons.jsonl", std::slice::from_ref(&l1));
3210        let trust = |b: Builder| b.ca_tls_config(iroh::tls::CaTlsConfig::insecure_skip_verify());
3211        let alice = std::sync::Arc::new(Node::open(&pa, None, true, trust).await.unwrap());
3212        let bob = Node::open(&pb, None, false, trust).await.unwrap().quiet();
3213        assert!(matches!(alice.relay(), Relay::Custom(_)));
3214        assert!(alice.tickets().is_empty() && bob.tickets().is_empty(), "no direct address on either side");
3215        let wait = std::time::Duration::from_secs(10);
3216        assert!(alice.online(wait).await && bob.online(wait).await, "both reach the relay");
3217        answer(alice.clone());
3218
3219        let mut t = ticket(&alice, "member", 60_000, 1);
3220        t.hints.retain(|h| !matches!(h, Hint::Direct(_)));
3221        let (m, rep) = bob.join(&t).await.unwrap();
3222        assert_eq!(m.role_of(&bob.public()), Some("member"));
3223        assert_eq!(rep.received, 1);
3224        assert!(holds(&home(&pb), "lessons.jsonl", &l1), "the lesson crossed through the relay");
3225        // And a later sync by key alone: the configured relay is the only way.
3226        let l2 = lesson("L2", 1, "and back?");
3227        append_lines(&home(&pb), "lessons.jsonl", std::slice::from_ref(&l2));
3228        assert_eq!(bob.sync(&alice.public()).await.unwrap().sent, 1);
3229        assert!(holds(&home(&pa), "lessons.jsonl", &l2));
3230        bob.close().await;
3231        alice.close().await;
3232    }
3233}