Crate dig_peer_selector

Expand description

§dig-peer-selector — the self-optimizing peer-selection middleware of the DIG Node

The selector is a pure decision + learning layer. It sits between dig-download (the executor that fetches bytes over dig-nat mTLS) and the peer-discovery layers (dig-dht, dig-gossip/dig-pex, dig-nat), and answers one question — “of these candidate peers, which subset should serve this content, and in what order?” — learning the answer from the real, measured outcome of every transfer it influenced. It has no user-facing configuration: every tradeoff (saturation point, relayed penalty, decay) is self-tuned from observed data.

This crate is the authoritative implementation of SPEC.md (version 1). The SPEC is the contract; this documentation summarizes it — read SPEC.md for the normative statements.

§The closed loop

The node calls dig_dht::DhtService::find_providers and maps the ProviderRecords into Candidates, then asks the selector to select the best subset for a ContentRequest.
dig-download executes the multi-source byte-range transfer over dig-nat mTLS mux streams.
Every per-range / per-request TransferOutcome streams back via record_outcome, updating the models in real time so the next select — and a mid-transfer rebalance — is smarter.

§What is learned (measured-only, non-gameable)

A peer’s quality is refined EXCLUSIVELY from measured outcomes (SPEC §3, §9.2): there is no input path by which a peer raises its own score, and observed capacity always overrides advertised (SPEC §9.3). Throughput/RTT are recency-weighted estimators whose decay is derived from each peer’s observed volatility — no baked constant (SPEC §3.2, §4.3). The scorer learns a per-class saturation point (anti-thundering-herd, SPEC §4.1) and an adaptive relayed penalty (SPEC §4.2), and orients toward minimizing P99 request latency (SPEC §4.4).

§Boundaries (what the selector is NOT)

It never queries the DHT, runs the gossip pool, opens a socket/TLS session/mux stream, or fetches/verifies/persists bytes — those belong to dig-dht, dig-gossip, dig-nat, and dig-download respectively (SPEC §1.2). The selector only reads their outputs or drives their choices. It re-uses the transport-verified PeerId (= SHA-256(TLS SPKI DER)) and the dig-dht content/candidate types verbatim — it defines no parallel identity (SPEC §5, §11).

§Implementers’ note — how `dig-node` embeds the selector (P3 integration, digstore)

The selector is the source-selection seam between dig-dht and dig-download. dig-node wires it as follows (the wiring lives in the node, NOT this crate — SPEC §6.1, §7.5):

Construct one PeerSelector per node: PeerSelector::new(SelectorConfig::default()).
Feed the registry continuously:
- subscribe to dig_gossip::GossipHandle::subscribe_pool_events() and forward each event — convert dig_gossip::PoolEvent → PoolEvent with a trivial 1:1 field map (identical shapes; see pool_event) — into on_pool_event;
- on each established dig-nat connection, call on_connection_class with its dig_nat::TraversalKind;
- optionally seed from a connected_pool_peers() snapshot at startup via upsert_candidate.
Per content want, call find_providers(&content), map each ProviderRecord into a Candidate (via Candidate::from_provider_record), and call select. Use each SelectedPeer’s max_concurrency as the per-peer concurrent-range cap in dig-download (replacing its built-in pick_source heuristic).
Translate the dig-download DownloadEvent stream into outcomes as it flows (SPEC §6.2): map RangeCompleted → a Range TransferOutcome with result: Success; RangeFailed → Failure { reason } (an integrity/verify failure → FailureReason::VerificationFailed); optionally Completed → a Request outcome for whole-request P99 learning. Call record_outcome for each. A Paused is NOT a failure — do not record one (SPEC §6.4).
On a dropped source / relocate, call rebalance with the still- active peers and the still-needed ranges to get a replacement subset (SPEC §5.5). On resume, select/rebalance only the ranges NOT in DownloadState::done_ranges (SPEC §6.4).

§DigPeer hand-off — the connect step, handed back

The selector returns ranked peer identities, and it still opens no socket. It does, however, hand back everything needed to open one: dial_plan pairs each SelectedPeer with the PeerTarget that reaches it, built from the addresses the registry learned for that peer. Pass the target straight to dig_peer::DigPeer::connect (or connect_with_runtime to compose the full NAT ladder) — the target carries the peer_id, which the handshake PINS, so a different CA-valid peer cannot answer in place of the chosen one (#1283).

Consumers MUST NOT re-derive this mapping from their own candidate lists: a second implementation of the addressing is how two crates silently drift apart on which address they dial. For the same reason the candidate ordering itself is inherited from dig_dht::dial_candidates — kind filter, host:port dedup, bound and reserved IPv4 fallback slot included — rather than restated here.

let selection = selector.select(&request, &candidates);
for (chosen, target) in selector.dial_plan(&selection, "mainnet") {
    let peer = dig_peer::DigPeer::connect(&target, &node_cert, &nat_config).await?;
    // ... fetch up to `chosen.max_concurrency` ranges from `peer`
}

Because TransferOutcome/Selection are defined here structurally, this crate does NOT depend on dig-download — avoiding a dependency cycle; the event→outcome mapping lives in the node adapter (SPEC §11).

Re-exports§

pub use config::ClockSource;
pub use config::SelectorConfig;
pub use config::DEFAULT_REGISTRY_CAPACITY;
pub use dial::peer_target;
pub use engine::PeerSelector;
pub use observe::peer_id_hex;
pub use observe::PeerSnapshot;
pub use observe::SelectorSnapshot;
pub use pool_event::PoolEvent;
pub use pool_event::PoolRemovalReason;
pub use quality::Estimate;
pub use quality::PeerQuality;
pub use quality::Reliability;
pub use registry::FeedResult;
pub use registry::PeerEntry;
pub use registry::DISPATCH_TTL_SECS;
pub use scoring::PeerClass;
pub use scoring::RelayModel;
pub use scoring::SaturationModel;
pub use types::Candidate;
pub use types::ContentRequest;
pub use types::FailureReason;
pub use types::OutcomeKind;
pub use types::OutcomeResult;
pub use types::Provenance;
pub use types::RangePlanDelta;
pub use types::SelectedPeer;
pub use types::Selection;
pub use types::TransferOutcome;

Modules§

config: SelectorConfig — wiring only, never behavior (SPEC.md §1.5, §5.6).
dial: Dial hand-off (SPEC §5.8, §11): turning a ranked identity into something a caller can actually connect to.
engine: PeerSelector — the public engine (SPEC.md §5): the closed select → record_outcome → rebalance decision + learning loop over the registry (§2), the measured-only quality model (§3), and the autonomous scorer (§4).
observe: Read-only observability snapshots (SPEC.md §5.7) — the agent-friendly / machine-consumable surface.
pool_event: PoolEvent / PoolRemovalReason — the topology/churn feed, byte-compatible with dig_gossip::PoolEvent (SPEC.md §5.4, §7.2).
quality: PeerQuality — the per-peer, measured-only capacity/RTT/reliability model (SPEC.md §3).
registry: The dynamic peer registry: peer_id -> PeerEntry, fed by churn + DHT candidates, bounded with lowest-value eviction (SPEC.md §2).
scoring: The autonomous scoring function (SPEC.md §4): learned saturation per class, an adaptive relayed penalty, volatility-driven decay (in crate::quality), the min-P99 + anti-thundering-herd objective, and the ranked-subset output — satisfying invariants A–G (SPEC §4.4).
types: The frozen public value types of the selector API (SPEC.md §5.1, §5.2, §5.5).

Structs§

CandidateAddr: The candidate/content types re-used from dig-dht (SPEC §7.1): what is fetched + how a provider is addressed. One candidate address for a provider: { host, port, kind } (L7 dig.getPeers §7). The finder dials these (most-direct-first) via dig_nat::connect to reach the provider.
PeerId: The transport-verified peer identity (peer_id = SHA-256(TLS SPKI DER)), re-used from dig-nat. A peer’s stable network identity: the 32-byte SHA-256 of its TLS SPKI DER.
PeerTarget: The dial target handed back for each selected peer, re-used from dig-peer (SPEC §5.8) — the peer_id-pinned input to dig_peer::DigPeer::connect. A description of the peer to connect to.
ProviderRecord: The candidate/content types re-used from dig-dht (SPEC §7.1): what is fetched + how a provider is addressed. The DHT’s stored value: peer provider_peer_id holds the content whose key is content_key, reachable at addresses, until expires_at.

Enums§

AddressKind: The candidate/content types re-used from dig-dht (SPEC §7.1): what is fetched + how a provider is addressed. How a candidate address was learned — the L7 dig.getPeers addresses[].kind tokens (§7). The lowercase serde spelling is the frozen wire form; the ordering is most-direct-first (a dialer picks the lowest-rank dialable candidate).
ContentId: The candidate/content types re-used from dig-dht (SPEC §7.1): what is fetched + how a provider is addressed. A DIG content identifier at store / root(capsule) / resource granularity — the key a provider record is stored under and a lookup asks for.
TraversalKind: The dig-nat connection-class ladder the selector reads observationally (SPEC §7.3). Which traversal technique produced a result — used to order methods, tag failures, and report (observability) which method actually succeeded WITHOUT the caller caring.

Constants§

MAX_DIAL_CANDIDATES: The bound on how many candidate addresses one dial target carries, re-used from dig-dht — the crate that owns the dial-candidate ordering this crate inherits (SPEC §5.8). Never re-declared here: a copied constant is a rival implementation waiting to drift. Upper bound on how many candidates dial_candidates hands a dialer for ONE peer, so a record padded with addresses cannot turn a single holder into a connect storm. Byte-for-byte the same bound dig-download applies on its own dial path, so a consumer that adopts this iterator sees no change in attempt count.