Expand description
Primary↔replica reconnect telemetry (issue #1243, PRD #1237 Phase B).
A replica drives a long-lived gRPC pull loop against its primary
(run_replica_loop). When that link drops — a failed pull_wal_records
or the initial connect retry — the loop persists the connecting
health state and keeps retrying. When a pull succeeds again the loop
persists healthy. A reconnect is exactly that transition: a link
that was up, went down, and came back up.
This module records that signal as a monotonic counter so red-ui can
explain instability (“this replica has reconnected 14 times in the last
hour”) instead of showing only a last-error snapshot. It is the
reconnect-counter producer/consumer slice of the operational telemetry
substrate (ADR 0060): measurement here, export through /metrics
(reddb_replication_reconnects_total) and the red-ui status read model.
§What is not counted
- The initial connect on startup. A replica coming up for the first time is not “reconnecting”; the counter only moves once the link has been healthy at least once and then recovers from a drop.
- Apply-side failures (
apply_error,divergence,relay_error,ack_error, …). Those are not link drops — the gRPC stream is still up — and have their own counters (reddb_replica_apply_errors_total). Treating them as drops would inflate the reconnect count, so only theconnectingstate marks the link down.
§Privacy (ADR 0060 §5)
Reconnect telemetry stores no endpoint, URL, credential, or
authorization material — only a monotonic count and the bounded
transition state. The exported series carries the node’s own stable
replica_id as its single dimension, never the primary’s address.
§Reset / restart behavior
The counter is an in-memory AtomicU64 scoped to the process lifetime.
It starts at 0 on every boot and is not persisted across restarts
— a process restart resets it to 0, exactly like the sibling
reddb_replication_full_resync_total / _partial_resync_total counters.
Prometheus treats a counter that drops to 0 as a reset (via the
process-start timestamp), so rate()/increase() stay correct across a
restart. red-ui reads the live value; it does not assume monotonicity
across a process boundary.
Structs§
- Replica
Link Metrics - Process-lifetime reconnect telemetry for the local replica’s link to its
primary. Lives on the runtime beside
super::logical::ReplicaApplyMetricsand is driven from the single health-persist chokepoint in the replica loop, so every link-state transition is observed exactly once.