Skip to main content

Module reconnect

Module reconnect 

Source
Expand description

Primary↔replica reconnect telemetry (issue #1243, PRD #1237 Phase B).

A replica drives a long-lived gRPC pull loop against its primary (run_replica_loop). When that link drops — a failed pull_wal_records or the initial connect retry — the loop persists the connecting health state and keeps retrying. When a pull succeeds again the loop persists healthy. A reconnect is exactly that transition: a link that was up, went down, and came back up.

This module records that signal as a monotonic counter so red-ui can explain instability (“this replica has reconnected 14 times in the last hour”) instead of showing only a last-error snapshot. It is the reconnect-counter producer/consumer slice of the operational telemetry substrate (ADR 0060): measurement here, export through /metrics (reddb_replication_reconnects_total) and the red-ui status read model.

§What is not counted

  • The initial connect on startup. A replica coming up for the first time is not “reconnecting”; the counter only moves once the link has been healthy at least once and then recovers from a drop.
  • Apply-side failures (apply_error, divergence, relay_error, ack_error, …). Those are not link drops — the gRPC stream is still up — and have their own counters (reddb_replica_apply_errors_total). Treating them as drops would inflate the reconnect count, so only the connecting state marks the link down.

§Privacy (ADR 0060 §5)

Reconnect telemetry stores no endpoint, URL, credential, or authorization material — only a monotonic count and the bounded transition state. The exported series carries the node’s own stable replica_id as its single dimension, never the primary’s address.

§Reset / restart behavior

The counter is an in-memory AtomicU64 scoped to the process lifetime. It starts at 0 on every boot and is not persisted across restarts — a process restart resets it to 0, exactly like the sibling reddb_replication_full_resync_total / _partial_resync_total counters. Prometheus treats a counter that drops to 0 as a reset (via the process-start timestamp), so rate()/increase() stay correct across a restart. red-ui reads the live value; it does not assume monotonicity across a process boundary.

Structs§

ReplicaLinkMetrics
Process-lifetime reconnect telemetry for the local replica’s link to its primary. Lives on the runtime beside super::logical::ReplicaApplyMetrics and is driven from the single health-persist chokepoint in the replica loop, so every link-state transition is observed exactly once.