indicatrix-worker 0.7.2

Coordinator and render worker for an indicatrix-backed design catalogue. `serve` (default build) is the coordinator: it serves indicatrix-net's read-only design-library protocol to viewers over mutual TLS and accepts joining render workers on a separate worker port. The optional `worker` feature adds render capacity -- `join` (dial a coordinator and render its requests), `serve --render` (render with the coordinator's own CPU/GPU) and `render` (a one-shot scene-to-PNG CLI).
Documentation
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
//! The coordinator's [`WorkerLane`]s: [`JoinedWorkerLane`] over a checked-out joined
//! worker connection and [`OwnLane`] over this machine's own tracer (`serve --render`).
//!
//! # How a joined lane holds its connection
//!
//! A job checks its workers out of the [`Registry`] when it starts (and later for every
//! worker that becomes available while it runs), and each
//! [`JoinedWorkerLane`] keeps its connection across chunks while they succeed back to
//! back. Holding per job rather than re-checking out per chunk means a lane never loses
//! its worker to another job between two chunks, and a connection's stream is always
//! drained to `DONE`/`ERROR` before anything else is written to it.
//!
//! Between two chunks the connection can genuinely sit idle for a while: a lane with
//! nothing left to claim waits (`indicatrix_dispatch::pool::epoch::Epoch::claim`) while
//! another lane's straggling chunk is still in flight, and the registry's own liveness
//! `PING` only reaches CHECKED-IN (idle-in-the-registry) connections -- a checked-out one
//! is invisible to it. Left alone, that combination lets a perfectly healthy connection
//! sit past the 45 s a joined worker hangs up at (`join::session::JOIN_IDLE_TIMEOUT`)
//! with no traffic at all, and the worker drops it out from under the job. Each
//! `JoinedWorkerLane` therefore runs its own small heartbeat thread
//! ([`held_ping_loop`]) for as long as it holds a connection idle between chunks,
//! `PING`ing it well before that deadline; `render_chunk` holds the same lock for the
//! whole duration of an actual request, so the heartbeat only ever touches a connection
//! that is genuinely between chunks, never one mid-request. The heartbeat itself only
//! holds that lock long enough to lift the connection out and (on a successful ping) put
//! it back, never across the `PING`/`PONG` round trip -- see
//! [`try_heartbeat_once`](self::try_heartbeat_once)'s doc comment for why that is still
//! never racing `render_chunk` over the same connection, at the cost of `render_chunk`
//! occasionally checking out a different idle worker instead of waiting the ping out.
//!
//! On any failed chunk the lane lets go before the pool's backoff pause: a broken stream
//! is [`WorkerHandle::discard`]ed (unregistered; `join` reconnects by itself), a worker
//! that merely refused the chunk is checked back in (idle, pinged again). The next chunk
//! then checks out ANY idle eligible worker -- possibly the same machine reconnected
//! under a new id -- so no idle connection is ever held through a pause. When the job
//! ends the lanes are dropped and every connection still held is checked back in.
//!
//! Lanes come and go while a job runs. A worker that registers (or becomes idle) while a
//! fan-out job is running is given a lane in it within one scheduler tick: the job's
//! `late::LateLanes` feed checks the idle eligible workers out (the same [`LaneNeed`] and
//! worker-count rules as at job start, and the same memory budget per lane) and the pool
//! starts a lane for each against the shared sample cursor. A lane whose connection broke
//! (a chunk or a heartbeat found it dead, [`WorkerLane::lost`]) is removed from the pool as
//! soon as its unfinished range is back on the cursor, as long as another lane remains; a
//! worker that reconnects under a new id is simply a new worker and gets a new lane the
//! same way. The one lane left in a pool is never removed: it keeps retrying (and may pick
//! up a replacement connection, subject to the job's [`LaneNeed`]) until the pool's failure
//! schedule retires it.
//!
//! # Chunk sizing across lanes of very different speed
//!
//! Each lane's next chunk is sized tail-aware and share-aware
//! (`indicatrix_dispatch::pool::epoch::Epoch::want` ->
//! `indicatrix_dispatch::ChunkPolicy::tail_aware_samples`): the smaller of the plain
//! target-duration chunk and this lane's proportional share (by rate) of the run's
//! outstanding samples. A fast joined worker (an A100 dialed in over `join`) can
//! therefore no longer be left idling in `Epoch::claim` while a much slower lane (the
//! coordinator's own GPU, say) works through an oversized last chunk it grabbed simply
//! because its own target-duration size happened to exceed what was left. The rates
//! that sizing (and the coordinator's own fastest-worker ranking for `Interactive`
//! requests) reads come from `super::job::Coordinator::rates`, a book shared
//! coordinator-wide across every viewer connection -- not rebuilt per connection -- so
//! a joined worker's calibration
//! survives a GUI export's successive one-shot connections and, keyed by its
//! certificate label, its own reconnects too. A live-view (`Interactive`) request
//! takes every idle joined worker besides the own lane by default
//! (`serve --interactive-workers all`); `--interactive-workers 0` serves it from the
//! own lane alone and `--interactive-workers <n>` caps it at the `n` fastest.
//!
//! A joined lane reports the rate of its tracing alone: the samples between the worker's
//! first and its last `PROGRESS` that advanced the count, over the time between those
//! two reports. The `FinalOnly` `FRAME` that follows the last advance (encode, upload,
//! decode) and the `DONE` after it are not part of the span, so a worker's rate does not
//! sink with the size of the picture it uploads and the proportional share it is given
//! stays in line with what it traces. A chunk too short for two advancing reports reports
//! first-report-to-`DONE` instead (see `chunk::rate_from_progress`).
//!
//! # HDR jobs
//!
//! Every lane of a job shares one [`JobLanes`]: what a replacement connection must
//! accept ([`LaneNeed`] -- the pixel count and, for an HDR scene, `hdr`), the map the
//! coordinator holds for the job, and the workers that refused that map. A joined
//! worker lacking the map answers the chunk's `RENDER_REQUEST` with `NEED_ASSET`; the
//! lane answers with `ASSET` from the held copy (see `chunk`). A worker that refuses the
//! map anyway (`ASSET_FAILED`, or `UNSUPPORTED_REQUEST`) costs the lane that one chunk:
//! the chunk returns to the pool and the worker is excluded from the rest of the job,
//! so it never counts toward retiring a lane more than once.

mod chunk;

pub use chunk::LaneTimeouts;
pub(in crate::coordinator) use chunk::{POLL, PatientReader, WRITE_TIMEOUT};

use super::registry::{Registry, WorkerHandle, WorkerInfo};
use crate::{assets::HeldAsset, cli::ComputeMode, validate::MAX_SAMPLES_PER_REQUEST};
use glam::Vec3;
use indicatrix::renderer::gpu_backend::GpuBackend;
use indicatrix_dispatch::{CancelToken, ChunkResult, SampleRange, WorkerLane};
use indicatrix_net::{
    SceneState,
    messages::{ClientMessage, StreamEvent, error_codes},
};
use std::{
    sync::{
        Arc, Mutex, PoisonError,
        atomic::{AtomicBool, AtomicU32, Ordering},
    },
    thread,
    time::{Duration, Instant},
};

/// What a joined worker must accept to take part in a job.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct LaneNeed {
    /// The job's `width * height`: the worker's `max_pixels` must cover it.
    pub pixels: u32,
    /// The scene is lit by an HDR map: the worker must advertise `hdr`.
    pub hdr: bool,
}

impl LaneNeed {
    /// A studio-lit job of `pixels` pixels.
    #[must_use]
    pub const fn pixels(pixels: u32) -> Self {
        Self { pixels, hdr: false }
    }

    /// Whether the worker `info` describes can take the job.
    #[must_use]
    pub const fn accepts(self, info: &WorkerInfo) -> bool {
        info.capability.max_pixels >= self.pixels && (!self.hdr || info.capability.hdr)
    }
}

/// One job's state shared by all its joined lanes (see the module doc comment).
pub struct JobLanes {
    need: LaneNeed,
    asset: Option<Arc<HeldAsset>>,
    timeouts: LaneTimeouts,
    /// Workers that refused this job's HDR map: never checked out again for it.
    excluded: Mutex<Vec<u32>>,
}

impl JobLanes {
    /// The shared state of a job needing `need`, holding `asset` (an HDR scene's map).
    #[must_use]
    pub const fn new(
        need: LaneNeed,
        asset: Option<Arc<HeldAsset>>,
        timeouts: LaneTimeouts,
    ) -> Self {
        Self {
            need,
            asset,
            timeouts,
            excluded: Mutex::new(Vec::new()),
        }
    }

    /// What a worker must accept to take part in this job.
    #[must_use]
    pub(in crate::coordinator) const fn need(&self) -> LaneNeed {
        self.need
    }

    /// Whether `info`'s worker can take a chunk of this job now.
    pub(in crate::coordinator) fn accepts(&self, info: &WorkerInfo) -> bool {
        self.need.accepts(info) && !self.lock_excluded().contains(&info.worker_id)
    }

    /// Keeps `worker_id` out of the rest of this job.
    fn exclude(&self, worker_id: u32) {
        let mut excluded = self.lock_excluded();
        if !excluded.contains(&worker_id) {
            excluded.push(worker_id);
        }
    }

    fn lock_excluded(&self) -> std::sync::MutexGuard<'_, Vec<u32>> {
        self.excluded.lock().unwrap_or_else(PoisonError::into_inner)
    }
}

/// Coordinator-local `request_id`s for requests sent to joined workers -- unique per
/// process, so a late event of an earlier chunk can never be mistaken for the current
/// chunk's.
static NEXT_WORKER_REQUEST_ID: AtomicU32 = AtomicU32::new(1);

/// A fresh coordinator-local request id.
pub(in crate::coordinator) fn next_worker_request_id() -> u32 {
    NEXT_WORKER_REQUEST_ID.fetch_add(1, Ordering::Relaxed)
}

/// A [`JoinedWorkerLane`]'s connection and when it last saw traffic, shared with its own
/// heartbeat thread ([`held_ping_loop`]) -- see the module doc comment.
struct Held {
    handle: Mutex<Option<WorkerHandle>>,
    last_traffic: Mutex<Instant>,
    /// The lane's connection broke (a chunk or a heartbeat found it dead) and no
    /// replacement has been checked out since: see [`WorkerLane::lost`].
    gone: AtomicBool,
}

/// How long a checked-out connection may go without traffic before a lane pings it
/// itself: comfortably under the 45 s a joined worker hangs up at
/// (`join::session::JOIN_IDLE_TIMEOUT`) and under the registry's own idle-ping cadence
/// (`liveness`'s 10 s), so a lane sitting in `Epoch::claim` waiting for other lanes to
/// finish never looks silent to the worker it is holding.
const HELD_PING_INTERVAL: Duration = Duration::from_secs(15);

/// How often [`held_ping_loop`] wakes to check -- short so [`JoinedWorkerLane`]'s `Drop`
/// never waits long for it to notice `stop`.
const HELD_PING_POLL: Duration = Duration::from_millis(200);

/// How long one heartbeat `PING` may take to get its `PONG`.
const HELD_PING_DEADLINE: Duration = Duration::from_secs(5);

/// A lane over one joined worker connection (see the module doc comment for how it
/// holds and replaces its connection).
pub struct JoinedWorkerLane {
    name: String,
    registry: Arc<Registry>,
    /// The job's shared lane state: what a replacement connection must accept, the
    /// held HDR map, the excluded workers.
    job: Arc<JobLanes>,
    held: Arc<Held>,
    heartbeat_stop: Arc<AtomicBool>,
    heartbeat: Option<thread::JoinHandle<()>>,
}

impl JoinedWorkerLane {
    /// A lane starting on the checked-out `handle`, for the job `job` describes. Spawns
    /// this lane's own heartbeat thread (see the module doc comment); a spawn failure
    /// (vanishingly unlikely) just means no heartbeat, not a construction failure --
    /// the ordinary liveness deadlines still apply, just without the extra margin.
    #[must_use]
    pub fn new(handle: WorkerHandle, registry: Arc<Registry>, job: Arc<JobLanes>) -> Self {
        let name = format!("joined worker #{}", handle.info().worker_id);
        let held = Arc::new(Held {
            handle: Mutex::new(Some(handle)),
            last_traffic: Mutex::new(Instant::now()),
            gone: AtomicBool::new(false),
        });
        let heartbeat_stop = Arc::new(AtomicBool::new(false));
        let heartbeat = {
            let held = Arc::clone(&held);
            let stop = Arc::clone(&heartbeat_stop);
            thread::Builder::new()
                .name(format!("{name}-heartbeat"))
                .spawn(move || held_ping_loop(&held, &stop))
                .ok()
        };
        Self {
            name,
            registry,
            job,
            held,
            heartbeat_stop,
            heartbeat,
        }
    }

    /// Runs `range` on `handle`, split into requests of at most
    /// [`MAX_SAMPLES_PER_REQUEST`] samples, summing their radiance; stops at the first
    /// short one. Also returns why the connection broke, if it did.
    fn run_on(
        &self,
        handle: &mut WorkerHandle,
        scene: &SceneState,
        range: SampleRange,
        cancel: &CancelToken,
    ) -> (ChunkResult, Option<String>) {
        let pixels = scene.width as usize * scene.height as usize;
        let worker_id = handle.info().worker_id;
        let mut total = ChunkResult::complete(vec![Vec3::ZERO; pixels], 0, None);
        let mut rest = range;
        let mut parts = 0;
        while !rest.is_empty() {
            let take = rest.samples.min(MAX_SAMPLES_PER_REQUEST);
            let request =
                chunk::worker_request(next_worker_request_id(), scene, rest.first_sample, take);
            let exchange = chunk::Exchange {
                worker_id,
                cancel,
                timeouts: self.job.timeouts,
                asset: self.job.asset.as_deref(),
            };
            let reply = chunk::run_request(handle.stream(), &request, exchange);
            parts += 1;
            if reply.assets_sent > 0 {
                self.registry.note_assets_forwarded(reply.assets_sent);
            }
            if self.job.need.hdr
                && matches!(
                    reply.worker_code,
                    Some(error_codes::ASSET_FAILED | error_codes::UNSUPPORTED_REQUEST)
                )
            {
                tracing::info!(
                    "coordinator job: worker #{worker_id} refused the job's HDR map; it takes no further \
                     chunk of this job"
                );
                self.job.exclude(worker_id);
            }
            let usable = if reply.done == take || reply.prefix {
                reply.done
            } else {
                0
            };
            if usable > 0 {
                for (acc, v) in total.sum.iter_mut().zip(&reply.sum) {
                    *acc += *v;
                }
                total.done += usable;
            }
            total.rate = if parts == 1 { reply.rate } else { None };
            if usable < take || reply.broken.is_some() {
                total.rate = None;
                total.error = reply
                    .error
                    .or_else(|| cancel.is_cancelled().then(|| "cancelled".to_string()))
                    .or_else(|| Some(format!("worker #{worker_id} ended the chunk early")));
                return (total, reply.broken);
            }
            rest = rest.after_prefix(take);
        }
        (total, None)
    }
}

impl WorkerLane for JoinedWorkerLane {
    fn name(&self) -> &str {
        &self.name
    }

    fn render_chunk(
        &self,
        scene: &SceneState,
        range: SampleRange,
        cancel: &CancelToken,
    ) -> ChunkResult {
        let mut slot = self
            .held
            .handle
            .lock()
            .unwrap_or_else(PoisonError::into_inner);
        let handle = slot
            .take()
            .or_else(|| Registry::checkout(&self.registry, |w| self.job.accepts(w)));
        let Some(mut handle) = handle else {
            self.held.gone.store(true, Ordering::Relaxed);
            return ChunkResult::failed("no idle joined worker to take the chunk".to_string());
        };
        self.held.gone.store(false, Ordering::Relaxed);
        let (result, broken) = self.run_on(&mut handle, scene, range, cancel);
        *self
            .held
            .last_traffic
            .lock()
            .unwrap_or_else(PoisonError::into_inner) = Instant::now();
        match broken {
            Some(why) => {
                self.held.gone.store(true, Ordering::Relaxed);
                handle.discard(&why);
            }
            // A clean chunk keeps the connection for the next one; a refused one goes
            // back to the registry before the pool's backoff pause.
            None if result.error.is_none() => *slot = Some(handle),
            None => drop(handle),
        }
        drop(slot);
        result
    }

    /// The connection broke, or no worker was left to carry on with: the pool removes
    /// the lane while others remain, and a worker that registers again (under a new id)
    /// is then given a lane of its own.
    fn lost(&self) -> bool {
        self.held.gone.load(Ordering::Relaxed)
    }
}

impl Drop for JoinedWorkerLane {
    /// Stops and joins this lane's heartbeat thread -- bounded by [`HELD_PING_POLL`],
    /// not [`HELD_PING_INTERVAL`], so ending a job never waits long per lane.
    fn drop(&mut self) {
        self.heartbeat_stop.store(true, Ordering::Relaxed);
        if let Some(thread) = self.heartbeat.take() {
            let _ = thread.join();
        }
    }
}

/// One [`JoinedWorkerLane`]'s heartbeat thread body (see the module doc comment): while
/// `stop` isn't set, every [`HELD_PING_INTERVAL`] with no other traffic, `PING`s the
/// connection currently idling in `held.handle`, if any.
///
/// `render_chunk` holds `held.handle`'s lock for the WHOLE duration of an actual
/// request, so a successful `try_lock` here, finding a connection in it, means the lane
/// is genuinely between chunks right now -- never mid-request, never racing
/// `render_chunk`'s own use of the same connection.
fn held_ping_loop(held: &Held, stop: &AtomicBool) {
    while !stop.load(Ordering::Relaxed) {
        thread::sleep(HELD_PING_POLL);
        if stop.load(Ordering::Relaxed) {
            return;
        }
        let due = held
            .last_traffic
            .lock()
            .unwrap_or_else(PoisonError::into_inner)
            .elapsed()
            >= HELD_PING_INTERVAL;
        if due {
            try_heartbeat_once(held);
        }
    }
}

/// One heartbeat attempt for [`held_ping_loop`]: pings the connection currently idling
/// in `held.handle`, if any, bumping `held.last_traffic` on success or discarding the
/// connection on failure. A no-op when nothing is checked out there right now, or when
/// `render_chunk` is already using it (`try_lock` finds it busy).
///
/// `held.handle`'s lock is taken only to lift the connection out and, on success, to put
/// it back -- never held across the ping itself (a network round trip): while it is out,
/// `render_chunk`'s own blocking `lock()` sees `None` and checks out a fresh idle worker
/// from the registry rather than waiting, which is fine -- this worker's registry slot
/// stays `CheckedOut` for as long as this `JoinedWorkerLane` exists regardless of whether
/// its handle currently sits in `held.handle` or in this function's local `handle`, so
/// nothing else can ever pick up the SAME connection out from under this ping.
fn try_heartbeat_once(held: &Held) {
    let mut slot = match held.handle.try_lock() {
        Ok(guard) => guard,
        Err(std::sync::TryLockError::Poisoned(poisoned)) => poisoned.into_inner(),
        Err(std::sync::TryLockError::WouldBlock) => {
            // `render_chunk` is using the connection right now: real traffic, no
            // heartbeat needed.
            return;
        }
    };
    let Some(mut handle) = slot.take() else {
        // Nothing checked out right now (lost earlier); `render_chunk` checks out
        // a replacement on its own next call.
        return;
    };
    drop(slot);
    match ping_held(&mut handle) {
        Ok(()) => {
            *held
                .last_traffic
                .lock()
                .unwrap_or_else(PoisonError::into_inner) = Instant::now();
            *held.handle.lock().unwrap_or_else(PoisonError::into_inner) = Some(handle);
        }
        Err(why) => {
            tracing::info!(
                "coordinator job: heartbeat ping to worker #{} failed while it idled between \
                 chunks: {why}",
                handle.info().worker_id
            );
            handle.discard(&why);
            held.gone.store(true, Ordering::Relaxed);
            // `held.handle` stays `None`; `render_chunk`'s own checkout picks a
            // replacement on its next call, unless the pool removes the lane first.
        }
    }
}

/// Sends `PING` on `handle`'s connection and waits (bounded by [`HELD_PING_DEADLINE`])
/// for the matching `PONG`.
fn ping_held(handle: &mut WorkerHandle) -> Result<(), String> {
    // A fixed nonce is fine: this exchange is fully synchronous, with `held.handle`'s
    // lock excluding `render_chunk` for its duration, so nothing else could be waiting
    // on a PONG of its own over this connection at the same time.
    const NONCE: u64 = 0x4845_4152_5442_4954; // "HEARTBIT" in ASCII hex, arbitrary.
    let mut conn = handle.stream();
    conn.set_timeouts(Some(HELD_PING_DEADLINE), Some(HELD_PING_DEADLINE))
        .map_err(|e| format!("could not arm the heartbeat ping deadline: {e}"))?;
    indicatrix_net::messages::write_message(&mut conn, &ClientMessage::Ping { nonce: NONCE })
        .map_err(|e| format!("heartbeat PING failed: {e}"))?;
    loop {
        let (event, _payload) = indicatrix_net::messages::read_stream_event(&mut conn)
            .map_err(|e| format!("heartbeat: no PONG ({e})"))?;
        match event {
            StreamEvent::Pong { nonce: NONCE } => {
                let _ = conn.set_timeouts(None, None);
                return Ok(());
            }
            other => tracing::debug!("coordinator: ignoring {other:?} from a held idle worker"),
        }
    }
}

/// The coordinator's own CPU/GPU lane (`serve --render`).
///
/// Each chunk runs the same tracer a streamed request uses
/// ([`crate::stream_emit::trace_range`]), the shared [`GpuBackend`] taking its FIFO turn
/// with every other request on this machine.
pub struct OwnLane {
    gpu: Arc<GpuBackend>,
    threads: usize,
    compute_mode: ComputeMode,
}

impl OwnLane {
    /// The own lane over `gpu` (disabled for `--only-cpu`) and `threads` CPU threads.
    #[must_use]
    pub const fn new(gpu: Arc<GpuBackend>, threads: usize, compute_mode: ComputeMode) -> Self {
        Self {
            gpu,
            threads,
            compute_mode,
        }
    }
}

impl WorkerLane for OwnLane {
    fn name(&self) -> &'static str {
        "coordinator's own lane"
    }

    fn render_chunk(
        &self,
        scene: &SceneState,
        range: SampleRange,
        cancel: &CancelToken,
    ) -> ChunkResult {
        let traced = crate::stream_emit::trace_range(
            scene,
            (range.first_sample, range.samples),
            self.threads,
            self.compute_mode,
            &self.gpu,
            cancel.as_flag(),
        );
        if traced.panicked {
            return ChunkResult::partial(
                traced.sum,
                traced.done,
                "the own lane's tracer panicked on this scene".to_string(),
            );
        }
        if traced.done < range.samples {
            return ChunkResult::partial(traced.sum, traced.done, "cancelled".to_string());
        }
        ChunkResult::complete(traced.sum, traced.done, None)
    }
}

#[cfg(test)]
mod tests {
    use super::*;
    use crate::coordinator::LivenessConfig;
    use indicatrix_net::messages::{Backend, PayloadEncoding, RenderCapability};
    use std::{
        net::{TcpListener, TcpStream},
        thread,
    };

    /// A checked-out [`WorkerHandle`] over a real loopback TCP pair, and the far end
    /// (kept open so the connection stays live).
    fn checked_out_handle() -> (WorkerHandle, TcpStream) {
        let registry = Registry::new(LivenessConfig::default());
        let listener = TcpListener::bind("127.0.0.1:0").expect("bind a loopback listener");
        let addr = listener
            .local_addr()
            .expect("a bound listener has a local address");
        let near = TcpStream::connect(addr).expect("connect to our own listener");
        let far = listener
            .accept()
            .expect("accept the connection just made")
            .0;
        let worker_id = registry.allocate_id();
        let info = WorkerInfo {
            worker_id,
            capability: RenderCapability {
                backend: Backend::Cpu { threads: 1 },
                max_pixels: 1_000,
                min_cadence_ms: 100,
                hdr: false,
            },
            peer: None,
            label: None,
            payload_encoding: PayloadEncoding::Raw,
        };
        registry.insert(info, Box::new(near), None);
        let handle =
            Registry::checkout(&registry, |_| true).expect("the just-inserted worker is idle");
        (handle, far)
    }

    /// A peer that reads exactly one `PING` and answers `PONG` with the same nonce,
    /// then stops (a stand-in for a joined worker's own request loop between chunks).
    fn answer_one_ping(mut far: TcpStream) {
        thread::spawn(move || {
            let message: ClientMessage = indicatrix_net::messages::read_message(&mut far)
                .expect("the heartbeat sends exactly one PING");
            let ClientMessage::Ping { nonce } = message else {
                panic!("expected ClientMessage::Ping, got {message:?}");
            };
            indicatrix_net::messages::write_stream_event(
                &mut far,
                &StreamEvent::Pong { nonce },
                None,
            )
            .expect("writing PONG must not fail on a live loopback socket");
        });
    }

    #[test]
    fn ping_held_succeeds_against_a_peer_that_answers_pong() {
        let (mut handle, far) = checked_out_handle();
        answer_one_ping(far);
        ping_held(&mut handle).expect("a live peer answering PONG must succeed");
    }

    #[test]
    fn ping_held_fails_against_a_silent_peer() {
        let (mut handle, _far) = checked_out_handle();
        // `_far` is kept alive (not dropped) but never answers -- `ping_held` must
        // still fail once `HELD_PING_DEADLINE` passes, not hang.
        assert!(ping_held(&mut handle).is_err());
    }
}