Skip to main content

sapphire_framework_session/
session.rs

1//! One replication session over one stream.
2//!
3//! Both sides run this same function. It is symmetric: each sends what the other lacks and
4//! applies what it is sent, and neither is a client.
5//!
6//! The exchange, in order:
7//!
8//! 1. `Hello` — format, workspace, replica id and the sender's version vector. A format
9//!    mismatch, or a different workspace, ends the session with `Refused` before any state
10//!    is exchanged.
11//! 2. Every path state the peer's version vector does not cover, in pages of [`PAGE`], each
12//!    page preceded by the content it inlines (64 KiB or less, and within the peer's
13//!    `max_file_size`), then `Done`.
14//! 3. `Want` for content that was not inlined, answered with the content or `Missing`. Once
15//!    the peer's `Done` is in, also `Want` for content earlier sessions never got
16//!    ([`Replica::unavailable`]).
17//! 4. `Settled`, once the sender will ask for nothing more.
18//! 5. `commit_session` last. An interrupted session leaves the version vector where it was,
19//!    and everything is simply resent next time. A completed one commits even when some
20//!    content was `Missing`: the paths are recorded, and step 3 of every later session asks
21//!    for their content again, from whichever peer it meets.
22//!
23//! # Locking and giving up
24//!
25//! The replica is locked per step, not across the exchange: to build this side's pages, to
26//! apply each page that arrives (and work out what it still needs), to answer each `Want`,
27//! and to materialise at the end. Content frames take no lock at all — they go straight to
28//! the staging directory — so a scan, or another session, runs while a large file is in
29//! flight. That is safe because `apply` joins path states, which commutes with a scan's own
30//! records, and committing only merges the peer's version vector.
31//!
32//! With one file able to take minutes, the exchange is bounded by silence, not by total
33//! time: it gives up once nothing has been read or written for [`IDLE_TIMEOUT`].
34//!
35//! The exchange is one function, [`initial_exchange`], because two callers start here:
36//! `run_session` finishes a session from what it returns, and `open_live_session` (in
37//! `live.rs`) keeps the stream and applies `Live` pushes on it.
38//!
39//! # Why there are two end markers
40//!
41//! `Done` says "my pages are complete" and is sent as soon as they are queued, so the peer
42//! can start asking for content without waiting for anything. That leaves a race worth
43//! naming: a peer that has just applied a page and is about to send `Want` looks exactly
44//! like a peer that needs nothing. A side that stopped reading at `Done` would therefore
45//! drop the `Want` for every file over [`INLINE_LIMIT`] — and, with it, the file. `Settled`
46//! closes the race. It is sent once the sender's want list can no longer grow (the peer's
47//! pages are in and every hash asked for has been answered), so the receiver knows no `Want`
48//! can follow, and both sides exit only after seeing the peer's `Done` *and* `Settled`.
49//!
50//! Content is addressed by its hash, so a page's inline blobs may travel in front of the page
51//! that references them. They do: a receiver that had already applied the page would
52//! otherwise ask for content that is on the wire, turning every inlined file into a `Want`
53//! and a second copy of its bytes.
54//!
55//! # Writers
56//!
57//! Everything this side sends goes through one task (`writer.rs`), so the read loop never
58//! blocks on a write, and its queues are unbounded, so queueing never waits on that task
59//! either. Two `run_session` calls facing each other would otherwise deadlock the moment
60//! both wrote more than the socket buffer, or filled a queue, while each waited to read.
61//!
62//! # Content
63//!
64//! Content is streamed from the file that holds it and received straight into the staging
65//! directory (`incoming.rs`), so neither side holds a whole file in memory. Applying a page
66//! finds content there, which is why `apply` and `fetch_missing` are given [`NoContent`].
67
68use std::collections::HashSet;
69
70use grain_id::GrainId;
71use sapphire_sync::{
72    Content, ContentHash, ContentSource, PathUpdate, Replica, Report, VersionVector,
73};
74use tokio::io::{AsyncRead, AsyncWrite, ReadHalf};
75use tokio::sync::Mutex;
76use tokio::task::JoinHandle;
77use tokio::time::Instant;
78
79use crate::error::{Error, Result};
80use crate::frame::{Frame, read_frame};
81use crate::incoming::Incoming;
82use crate::message::Message;
83use crate::progress::Progress;
84use crate::writer::{Out, Outbox, spawn_writer};
85use crate::{IDLE_TIMEOUT, INLINE_LIMIT, SESSION_FORMAT_VERSION};
86
87/// How many path states travel in one `Updates` message.
88const PAGE: usize = 256;
89
90/// How long the exchange waits for the peer's `Hello`.
91///
92/// The one wait in a session with nothing behind it. Every later read is answering something
93/// the peer asked for or was told, and a peer that stops mid-exchange ends the read with end
94/// of file. The `Hello` is different: the stream is open, the peer has its end, and it may
95/// simply never speak — a bridge that parked the stream for an app server which never
96/// claimed it, or a peer that went away between the dial and its first frame on a transport
97/// with no half-close to report it with.
98///
99/// Waiting for ever is not an option: one silent peer would pin a session, its stream and
100/// whatever drives it for the rest of the process's life. The same reasoning
101/// [`IDLE_TIMEOUT`] applies to the rest of the exchange.
102///
103/// Generous on purpose, and a bound rather than a pace: a peer's bridge may hold a stream
104/// parked for about a minute while it starts a stopped app server, which is waiting that is
105/// working as intended, and nothing waits this out when the peer answers.
106pub const HELLO_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(90);
107
108/// What a session did.
109#[derive(Clone, Debug, Default)]
110pub struct SessionOutcome {
111    /// Path updates sent.
112    pub sent: usize,
113    /// Path updates received.
114    pub received: usize,
115    /// What applying them did.
116    pub report: Report,
117}
118
119/// The content source a session hands the replica: none.
120///
121/// Received content is already in the staging directory by the time a page is applied,
122/// and the replica looks there first.
123pub(crate) struct NoContent;
124
125impl ContentSource for NoContent {
126    fn locate(&self, _: &ContentHash) -> Option<std::path::PathBuf> {
127        None
128    }
129}
130
131/// Why the read loop stopped.
132pub(crate) enum Stop {
133    /// The peer's `Done` and `Settled` are in and this side wants nothing more.
134    Complete,
135    /// The stream ended; the peer is gone.
136    Gone,
137    /// The peer refused to continue, and why.
138    Refused(String),
139    /// The peer sent something this session does not allow.
140    Protocol(String),
141}
142
143/// Everything the initial exchange left behind, for the caller that decides what happens
144/// next.
145pub(crate) struct Exchange<S> {
146    /// The peer's half of the stream, still to be read.
147    pub(crate) reader: ReadHalf<S>,
148    /// Everything this side sends goes through here, and one task writes it.
149    pub(crate) out: Outbox,
150    /// The task owning the write half.
151    pub(crate) writer: JoinHandle<()>,
152    /// When the session last moved bytes, marked by the writer task and the read loop.
153    pub(crate) progress: Progress,
154    /// Everything the peer said it has, as of its `Hello`.
155    pub(crate) peer_vv: VersionVector,
156    /// Whether the peer's `Hello` said it answers a live push with `Ack`.
157    pub(crate) peer_acks: bool,
158    /// Everything this side had when it worked out what to send: the peer has all of it
159    /// once the exchange completes.
160    pub(crate) sent_vv: VersionVector,
161    /// The content being received, if any, carried on into the live phase.
162    pub(crate) incoming: Incoming,
163    /// What the exchange did.
164    pub(crate) outcome: SessionOutcome,
165    /// How the read loop stopped.
166    pub(crate) stop: Stop,
167}
168
169/// Run a session to completion.
170///
171/// Returns once both sides have said `Done` and `Settled` and every piece of content this
172/// side asked for has been answered — with the bytes, or with `Missing`. A stream that ends
173/// first returns without an error, but commits nothing.
174///
175/// `replica` is locked for each step that touches it, never across a wait on the peer — not
176/// for its `Hello` (issue #163), and not while content is in flight. A caller that used to
177/// pre-lock a `Mutex<Replica>` around the whole call should stop: passing the mutex itself
178/// is what leaves the replica free for a scan or another session meanwhile.
179///
180/// A peer that stops making progress is given up on after [`IDLE_TIMEOUT`] with
181/// [`Error::Protocol`]; nothing bounds the session's total length.
182///
183/// `S` must be `'static` because the write half is moved into a task; a stream owned by the
184/// caller (a socket, a spliced pipe, a duplex) is.
185pub async fn run_session<S>(
186    stream: S,
187    replica: &Mutex<Replica>,
188    workspace_id: GrainId,
189) -> Result<SessionOutcome>
190where
191    S: AsyncRead + AsyncWrite + Send + Unpin + 'static,
192{
193    let Exchange {
194        out,
195        writer,
196        progress,
197        peer_vv,
198        outcome,
199        stop,
200        ..
201    } = initial_exchange(stream, replica, workspace_id).await?;
202    match stop {
203        Stop::Complete => {
204            // The peer is still reading: it exits only once it has seen this side's
205            // `Settled`, which is queued but not yet written. Draining the writer delivers
206            // `Done`, `Settled` and every pending answer.
207            let result = {
208                let mut locked = replica.lock().await;
209                materialise(&mut locked, &peer_vv, outcome)
210            };
211            close(out, writer, &progress, result).await
212        }
213        // The peer is gone or has misbehaved, so there is nothing to deliver and a write
214        // may never complete. Abort: nothing is committed either way, and everything is
215        // resent next session.
216        Stop::Gone => {
217            abort(out, writer);
218            Ok(outcome)
219        }
220        Stop::Refused(why) => {
221            abort(out, writer);
222            Err(Error::Refused(why))
223        }
224        Stop::Protocol(why) => {
225            abort(out, writer);
226            Err(Error::Protocol(why))
227        }
228    }
229}
230
231/// Run the initial exchange: `Hello`, this side's pages, and reading until the peer's
232/// `Done` and `Settled` are in and every hash this side asked for has been answered.
233///
234/// Both session kinds start here. Nothing is committed: the caller decides, so that an
235/// interrupted exchange leaves the version vector where it was. An error means the exchange
236/// never reached its end markers, and `run_session`'s own error mapping is the caller's job.
237///
238/// `replica` is locked per step, never across a wait on the peer: briefly at the very start,
239/// to read the two values this side's own `Hello` needs (issue #163); once the peer's `Hello`
240/// is in, to build this side's pages; then for each page that arrives (`apply` and
241/// `needed`) and each `Want` (`content_path`). Content frames go to staging and take no
242/// lock, so a scan or another session gets through while a large file is in flight.
243///
244/// After the `Hello`, the read loop gives up once nothing has been read or written for
245/// [`IDLE_TIMEOUT`].
246pub(crate) async fn initial_exchange<S>(
247    stream: S,
248    replica: &Mutex<Replica>,
249    workspace_id: GrainId,
250) -> Result<Exchange<S>>
251where
252    S: AsyncRead + AsyncWrite + Send + Unpin + 'static,
253{
254    let (mut reader, writer) = tokio::io::split(stream);
255
256    // The writer task owns the write half, so the read loop is never blocked by a peer that
257    // is slow to read. It marks `progress` as it writes, and the read loop as it reads.
258    let progress = Progress::new();
259    let (out_tx, writer_task) = spawn_writer(writer, progress.clone());
260
261    // 1. Hello, before anything else. Reading `replica_id`/`vv` is instantaneous, so the
262    // lock is released again immediately — nothing below here needs the replica until the
263    // peer's own `Hello` is in.
264    let (replica_id, vv, max_file_size) = {
265        let locked = replica.lock().await;
266        (
267            locked.replica_id(),
268            locked.vv().clone(),
269            locked.config().max_file_size,
270        )
271    };
272    send(
273        &out_tx,
274        Out::Control(Message::Hello {
275            format: SESSION_FORMAT_VERSION,
276            workspace_id,
277            replica_id,
278            vv,
279            max_file_size,
280            acks: true,
281        }),
282    )?;
283
284    // A peer that never speaks is given up on rather than waited out: see [`HELLO_TIMEOUT`].
285    // The queue is dropped rather than drained, because a peer that did not send a `Hello`
286    // cannot be expected to be reading either.
287    let hello = match tokio::time::timeout(HELLO_TIMEOUT, read_frame(&mut reader)).await {
288        // Progress like any other frame read: the idle clock starts here, not at the
289        // dial, so a `Hello` that took its time does not count against `IDLE_TIMEOUT`.
290        Ok(frame) => {
291            progress.touch();
292            frame
293        }
294        Err(_) => {
295            abort(out_tx, writer_task);
296            return Err(Error::Protocol(format!(
297                "the peer sent no Hello within {HELLO_TIMEOUT:?}"
298            )));
299        }
300    };
301
302    let (peer_vv, peer_max, peer_acks) = match hello? {
303        Some(Frame::Control(Message::Hello {
304            format,
305            workspace_id: theirs,
306            vv,
307            max_file_size,
308            acks,
309            ..
310        })) => {
311            if format != SESSION_FORMAT_VERSION {
312                let _ = send(
313                    &out_tx,
314                    Out::Control(Message::Refused(format!(
315                        "session format {format}, expected {SESSION_FORMAT_VERSION}"
316                    ))),
317                );
318                return refuse(
319                    out_tx,
320                    writer_task,
321                    Error::VersionMismatch {
322                        ours: SESSION_FORMAT_VERSION,
323                        theirs: format,
324                    },
325                )
326                .await;
327            }
328            if theirs != workspace_id {
329                let _ = send(
330                    &out_tx,
331                    Out::Control(Message::Refused(format!(
332                        "this stream is for workspace {workspace_id}, not {theirs}"
333                    ))),
334                );
335                return refuse(
336                    out_tx,
337                    writer_task,
338                    Error::Protocol(format!(
339                        "workspace mismatch: the peer is syncing {theirs}, this session is for \
340                     {workspace_id}"
341                    )),
342                )
343                .await;
344            }
345            (vv, max_file_size, acks)
346        }
347        Some(Frame::Control(Message::Refused(why))) => {
348            return refuse(out_tx, writer_task, Error::Refused(why)).await;
349        }
350        other => {
351            return refuse(
352                out_tx,
353                writer_task,
354                Error::Protocol(format!("expected Hello, got {other:?}")),
355            )
356            .await;
357        }
358    };
359
360    // 2. Build the outgoing side as owned data, then hand it to a task. Computing it up
361    //    front, in one short hold of the lock, keeps the read loop free while it is written.
362    let (mut incoming, sent_vv, sent, outgoing) = {
363        let replica = replica.lock().await;
364        let incoming = Incoming::new(replica.stager());
365        let delta = replica
366            .delta_for(&peer_vv)
367            .map_err(|e| Error::Sync(e.to_string()))?;
368        // Small content goes inline, ahead of the page that needs it — but none over the
369        // peer's size limit, which it would refuse. Looked up in one batch: a lookup per
370        // hash would rebuild the store index each time.
371        let inline_max = (INLINE_LIMIT as u64).min(peer_max);
372        let inline = replica
373            .content_paths(&inline_hashes(&delta, inline_max))
374            .unwrap_or_default();
375        let mut outgoing: Vec<Out> = Vec::new();
376        for page in delta.chunks(PAGE) {
377            for hash in inline_hashes(page, inline_max) {
378                if let Some((path, len)) = inline.get(&hash) {
379                    outgoing.push(Out::Blob {
380                        hash,
381                        path: path.clone(),
382                        len: *len,
383                    });
384                }
385            }
386            outgoing.push(Out::Control(Message::Updates(page.to_vec())));
387        }
388        outgoing.push(Out::Control(Message::Done));
389        (incoming, replica.vv().clone(), delta.len(), outgoing)
390    };
391
392    // 3. Read until the peer is done and settled and every hash we asked for has been
393    //    answered.
394    let mut wanted: HashSet<ContentHash> = HashSet::new();
395    // Hashes the peer answered without the content this session; not asked for again.
396    let mut lacked: HashSet<ContentHash> = HashSet::new();
397    let mut report = Report::default();
398    let mut received_count = 0usize;
399    let mut peer_done = false;
400    let mut peer_settled = false;
401    let mut settled = false;
402
403    // Every error out of the loop comes out of this block, so that one place below takes the
404    // writer down with it: the peer cannot be expected to read what is still queued.
405    let looped: Result<Stop> = async {
406        for item in outgoing {
407            send(&out_tx, item)?;
408        }
409        Ok(loop {
410            if peer_done && peer_settled && wanted.is_empty() {
411                break Stop::Complete;
412            }
413            let Some(frame) = read_or_idle(&mut reader, &progress).await? else {
414                break Stop::Gone;
415            };
416            match frame {
417                Frame::Control(Message::Updates(updates)) => {
418                    received_count += updates.len();
419                    // The join is idempotent, so a redelivered page costs nothing.
420                    let need = {
421                        let mut replica = replica.lock().await;
422                        let page = replica
423                            .apply(&updates, &NoContent)
424                            .map_err(|e| Error::Sync(e.to_string()))?;
425                        merge_report(&mut report, page);
426                        needed(&replica, &updates)?
427                    };
428                    // Ask for content the page needs that this session does not have.
429                    for hash in need {
430                        if wanted.insert(hash) {
431                            send(&out_tx, Out::Control(Message::Want(hash)))?;
432                        }
433                    }
434                }
435                Frame::Control(Message::Done) => {
436                    peer_done = true;
437                    // Content an earlier session never got: its paths were committed all the
438                    // same, so no page carries them again (#297). Asked for once the peer's
439                    // pages and the content they inline are all in, so nothing on the wire is
440                    // asked for twice.
441                    let again = replica
442                        .lock()
443                        .await
444                        .unavailable()
445                        .map_err(|e| Error::Sync(e.to_string()))?;
446                    for hash in again {
447                        if !lacked.contains(&hash) && wanted.insert(hash) {
448                            send(&out_tx, Out::Control(Message::Want(hash)))?;
449                        }
450                    }
451                }
452                Frame::Control(Message::Settled) => peer_settled = true,
453                Frame::Control(Message::Want(hash)) => {
454                    let reply = answer(&*replica.lock().await, hash);
455                    send(&out_tx, reply)?;
456                }
457                Frame::Control(Message::Missing(hash)) => {
458                    // The peer does not have it either. Another peer may: the path stays
459                    // unmaterialised, and later sessions ask for it again.
460                    wanted.remove(&hash);
461                    lacked.insert(hash);
462                }
463                Frame::Control(Message::Aborted(hash)) => {
464                    // The peer's file changed under it: as good as `Missing`.
465                    if let Err(e) = incoming.aborted(hash) {
466                        break Stop::Protocol(reason(e));
467                    }
468                    wanted.remove(&hash);
469                    lacked.insert(hash);
470                }
471                Frame::Control(Message::Live(_)) => {
472                    // A `Live` push travels only after both sides have left the exchange, and
473                    // frames from one peer arrive in order, so a compliant peer cannot send one
474                    // here.
475                    break Stop::Protocol(
476                        "a Live push before the initial exchange finished".to_owned(),
477                    );
478                }
479                Frame::Control(Message::Ack(_)) => {
480                    // An `Ack` answers a `Live` push, so it cannot come before one either.
481                    break Stop::Protocol("an Ack before the initial exchange finished".to_owned());
482                }
483                Frame::Control(Message::Refused(why)) => {
484                    break Stop::Refused(why);
485                }
486                Frame::Control(Message::Hello { .. }) => {
487                    break Stop::Protocol("a second Hello".to_owned());
488                }
489                // Content is verified before it is staged: it is addressed by hash, so a peer that
490                // sent the wrong bytes under one must not be able to plant them.
491                Frame::BlobStart { hash, len } => {
492                    if let Err(e) = incoming.start(hash, len) {
493                        break Stop::Protocol(reason(e));
494                    }
495                }
496                Frame::BlobChunk(bytes) => {
497                    if let Err(e) = incoming.chunk(&bytes) {
498                        break Stop::Protocol(reason(e));
499                    }
500                }
501                Frame::BlobEnd => match incoming.end() {
502                    Ok(hash) => {
503                        wanted.remove(&hash);
504                    }
505                    Err(e) => break Stop::Protocol(reason(e)),
506                },
507            }
508            // The peer's pages are all in, so this side's want list can no longer grow: settling
509            // tells the peer it may close as soon as it wants nothing more either.
510            if peer_done && wanted.is_empty() && !settled {
511                settled = true;
512                send(&out_tx, Out::Control(Message::Settled))?;
513            }
514        })
515    }
516    .await;
517    let stop = match looped {
518        Ok(stop) => stop,
519        // The writer failed to write, so the stream is gone: the same end as the read side
520        // seeing it close, whichever of the two notices first.
521        Err(Error::Closed) => Stop::Gone,
522        Err(e) => {
523            abort(out_tx, writer_task);
524            return Err(e);
525        }
526    };
527
528    Ok(Exchange {
529        sent_vv,
530        reader,
531        out: out_tx,
532        writer: writer_task,
533        progress,
534        peer_vv,
535        peer_acks,
536        incoming,
537        outcome: SessionOutcome {
538            sent,
539            received: received_count,
540            report,
541        },
542        stop,
543    })
544}
545
546/// Read the peer's next frame, or fail once the session has made no progress — nothing
547/// read and nothing written — for [`IDLE_TIMEOUT`].
548///
549/// The writer task marks `progress` as it writes, so a side busy sending a large file is
550/// not cut off for hearing nothing back. Each frame read marks it too.
551pub(crate) async fn read_or_idle<R: AsyncRead + Unpin>(
552    r: &mut R,
553    progress: &Progress,
554) -> Result<Option<Frame>> {
555    // `read_frame` is not cancel-safe: dropping it mid-frame would lose the bytes it has
556    // read. One pinned future survives every re-armed deadline below.
557    let read = read_frame(r);
558    tokio::pin!(read);
559    loop {
560        let deadline = Instant::now() + IDLE_TIMEOUT.saturating_sub(progress.idle_for());
561        tokio::select! {
562            frame = &mut read => {
563                progress.touch();
564                return frame;
565            }
566            // The writer may have moved since the deadline was set; only a full
567            // `IDLE_TIMEOUT` of nothing ends the session.
568            _ = tokio::time::sleep_until(deadline) => {
569                if progress.idle_for() >= IDLE_TIMEOUT {
570                    return Err(Error::Protocol(format!("no progress for {IDLE_TIMEOUT:?}")));
571                }
572            }
573        }
574    }
575}
576
577/// Materialise what was waiting on content, then commit.
578///
579/// Committing is what makes this replica claim it has what the peer had. It runs last, and
580/// only when the peer said `Done` and `Settled`; an interrupted session leaves the version
581/// vector where it was.
582pub(crate) fn materialise(
583    replica: &mut Replica,
584    peer_vv: &VersionVector,
585    mut outcome: SessionOutcome,
586) -> Result<SessionOutcome> {
587    let fetched = replica
588        .fetch_missing(&NoContent)
589        .map_err(|e| Error::Sync(e.to_string()))?;
590    merge_report(&mut outcome.report, fetched);
591    replica
592        .commit_session(peer_vv)
593        .map_err(|e| Error::Sync(e.to_string()))?;
594    tracing::debug!(
595        sent = outcome.sent,
596        received = outcome.received,
597        changed = outcome.report.changed,
598        "session complete"
599    );
600    Ok(outcome)
601}
602
603/// Queue one frame, or report that the writer is gone.
604pub(crate) fn send(out: &Outbox, item: Out) -> Result<()> {
605    out.send(item)
606}
607
608/// The answer to the peer's `Want(hash)`: the content, streamed from the file that holds
609/// it, or `Missing`.
610pub(crate) fn answer(replica: &Replica, hash: ContentHash) -> Out {
611    match replica.content_path(&hash) {
612        Ok(Some((path, len))) => Out::Blob { hash, path, len },
613        _ => Out::Control(Message::Missing(hash)),
614    }
615}
616
617/// What a [`Stop::Protocol`] says, from the error that stopped the read loop.
618pub(crate) fn reason(err: Error) -> String {
619    match err {
620        Error::Protocol(why) => why,
621        other => other.to_string(),
622    }
623}
624
625/// Close the outgoing side, let the writer flush what is queued, and return `result`.
626///
627/// Dropping the sender is what ends the writer task; it drains first, so a `Refused` or a
628/// `Settled` queued moments ago still reaches the peer before the stream goes away.
629///
630/// The wait is bounded like the exchange: a peer that holds the stream open but stops
631/// reading would otherwise keep the writer, and this call, waiting for ever. Once the
632/// writer has moved nothing for [`IDLE_TIMEOUT`] it is aborted; `result` stands either
633/// way, since the session's work is already done.
634async fn close(
635    out_tx: Outbox,
636    mut writer: JoinHandle<()>,
637    progress: &Progress,
638    result: Result<SessionOutcome>,
639) -> Result<SessionOutcome> {
640    drop(out_tx);
641    loop {
642        let deadline = Instant::now() + IDLE_TIMEOUT.saturating_sub(progress.idle_for());
643        tokio::select! {
644            _ = &mut writer => break,
645            // The writer may have moved since the deadline was set; only a full
646            // `IDLE_TIMEOUT` of nothing gives up on it.
647            _ = tokio::time::sleep_until(deadline) => {
648                if progress.idle_for() >= IDLE_TIMEOUT {
649                    tracing::debug!("the peer stopped reading; dropping what was still queued");
650                    writer.abort();
651                    break;
652                }
653            }
654        }
655    }
656    result
657}
658
659/// Send a `Refused` that is already queued, close the writer, and return `err`.
660///
661/// The exchange's own early exits: the peer is still reading, so what was queued has to
662/// reach it before the stream goes away.
663async fn refuse<S>(out_tx: Outbox, writer: JoinHandle<()>, err: Error) -> Result<Exchange<S>> {
664    drop(out_tx);
665    let _ = writer.await;
666    Err(err)
667}
668
669/// Close the outgoing side without waiting to deliver what is queued.
670///
671/// Used only once the peer can no longer be expected to read, where a write may otherwise
672/// never complete.
673pub(crate) fn abort(out_tx: Outbox, writer: JoinHandle<()>) {
674    drop(out_tx);
675    writer.abort();
676}
677
678/// The hashes of the content in `updates` of `max` bytes or less, to send inline.
679fn inline_hashes(updates: &[PathUpdate], max: u64) -> Vec<ContentHash> {
680    updates
681        .iter()
682        .flat_map(|update| &update.versions)
683        .filter_map(|entry| match entry.content {
684            Content::File { hash, len } if len <= max => Some(hash),
685            _ => None,
686        })
687        .collect()
688}
689
690/// Hashes these updates need that neither a file on disk nor the staging directory holds.
691///
692/// Content over this replica's size limit is never needed: it is recorded but not
693/// written, and the peer's `BLOB_START` for it would be refused as a protocol error.
694pub(crate) fn needed(replica: &Replica, updates: &[PathUpdate]) -> Result<Vec<ContentHash>> {
695    let max = replica.config().max_file_size;
696    let hashes: Vec<ContentHash> = updates
697        .iter()
698        .flat_map(|update| &update.versions)
699        .filter_map(|entry| match entry.content {
700            Content::File { hash, len } if len <= max => Some(hash),
701            _ => None,
702        })
703        .collect();
704    // One batch per page: a lookup per hash would rebuild the store index each time.
705    let have = replica
706        .content_paths(&hashes)
707        .map_err(|e| Error::Sync(e.to_string()))?;
708    Ok(hashes
709        .into_iter()
710        .filter(|hash| !have.contains_key(hash))
711        .collect())
712}
713
714pub(crate) fn merge_report(into: &mut Report, from: Report) {
715    into.changed += from.changed;
716    into.recorded.extend(from.recorded);
717    into.conflicts.extend(from.conflicts);
718    into.skipped.extend(from.skipped);
719}