Skip to main content

Crate dig_peer_selector

Crate dig_peer_selector 

Source
Expand description

§dig-peer-selector — the self-optimizing peer-selection middleware of the DIG Node

The selector is a pure decision + learning layer. It sits between dig-download (the executor that fetches bytes over dig-nat mTLS) and the peer-discovery layers (dig-dht, dig-gossip/dig-pex, dig-nat), and answers one question — “of these candidate peers, which subset should serve this content, and in what order?” — learning the answer from the real, measured outcome of every transfer it influenced. It has no user-facing configuration: every tradeoff (saturation point, relayed penalty, decay) is self-tuned from observed data.

This crate is the authoritative implementation of SPEC.md (version 1). The SPEC is the contract; this documentation summarizes it — read SPEC.md for the normative statements.

§The closed loop

  1. The node calls dig_dht::DhtService::find_providers and maps the ProviderRecords into Candidates, then asks the selector to select the best subset for a ContentRequest.
  2. dig-download executes the multi-source byte-range transfer over dig-nat mTLS mux streams.
  3. Every per-range / per-request TransferOutcome streams back via record_outcome, updating the models in real time so the next select — and a mid-transfer rebalance — is smarter.

§What is learned (measured-only, non-gameable)

A peer’s quality is refined EXCLUSIVELY from measured outcomes (SPEC §3, §9.2): there is no input path by which a peer raises its own score, and observed capacity always overrides advertised (SPEC §9.3). Throughput/RTT are recency-weighted estimators whose decay is derived from each peer’s observed volatility — no baked constant (SPEC §3.2, §4.3). The scorer learns a per-class saturation point (anti-thundering-herd, SPEC §4.1) and an adaptive relayed penalty (SPEC §4.2), and orients toward minimizing P99 request latency (SPEC §4.4).

§Boundaries (what the selector is NOT)

It never queries the DHT, runs the gossip pool, opens a socket/TLS session/mux stream, or fetches/verifies/persists bytes — those belong to dig-dht, dig-gossip, dig-nat, and dig-download respectively (SPEC §1.2). The selector only reads their outputs or drives their choices. It re-uses the transport-verified PeerId (= SHA-256(TLS SPKI DER)) and the dig-dht content/candidate types verbatim — it defines no parallel identity (SPEC §5, §11).

§Implementers’ note — how dig-node embeds the selector (P3 integration, digstore)

The selector is the source-selection seam between dig-dht and dig-download. dig-node wires it as follows (the wiring lives in the node, NOT this crate — SPEC §6.1, §7.5):

  1. Construct one PeerSelector per node: PeerSelector::new(SelectorConfig::default()).
  2. Feed the registry continuously:
  3. Per content want, call find_providers(&content), map each ProviderRecord into a Candidate (via Candidate::from_provider_record), and call select. Use each SelectedPeer’s max_concurrency as the per-peer concurrent-range cap in dig-download (replacing its built-in pick_source heuristic).
  4. Translate the dig-download DownloadEvent stream into outcomes as it flows (SPEC §6.2): map RangeCompleted → a Range TransferOutcome with result: Success; RangeFailed → Failure { reason } (an integrity/verify failure → FailureReason::VerificationFailed); optionally Completed → a Request outcome for whole-request P99 learning. Call record_outcome for each. A Paused is NOT a failure — do not record one (SPEC §6.4).
  5. On a dropped source / relocate, call rebalance with the still- active peers and the still-needed ranges to get a replacement subset (SPEC §5.5). On resume, select/rebalance only the ranges NOT in DownloadState::done_ranges (SPEC §6.4).

§DigPeer hand-off — how consumers establish connections

The selector returns ranked peer identities (each SelectedPeer carries a PeerId); it does NOT establish connections. The node is responsible for connecting to each selected peer via the dig-peer crate: construct a dig_peer::PeerTarget from the peer_id + addresses, then call dig_peer::DigPeer::connect to establish mTLS and get a dig_peer::Connected for the actual streams (SPEC §6.1, §11). The selector’s role ends at selection; the transport is entirely the node’s (and dig-download’s via dig-nat) responsibility.

Because TransferOutcome/Selection are defined here structurally, this crate does NOT depend on dig-download — avoiding a dependency cycle; the event→outcome mapping lives in the node adapter (SPEC §11).

Re-exports§

pub use config::ClockSource;
pub use config::SelectorConfig;
pub use config::DEFAULT_REGISTRY_CAPACITY;
pub use engine::PeerSelector;
pub use observe::peer_id_hex;
pub use observe::PeerSnapshot;
pub use observe::SelectorSnapshot;
pub use pool_event::PoolEvent;
pub use pool_event::PoolRemovalReason;
pub use quality::Estimate;
pub use quality::PeerQuality;
pub use quality::Reliability;
pub use registry::FeedResult;
pub use registry::PeerEntry;
pub use registry::DISPATCH_TTL_SECS;
pub use scoring::PeerClass;
pub use scoring::RelayModel;
pub use scoring::SaturationModel;
pub use types::Candidate;
pub use types::ContentRequest;
pub use types::FailureReason;
pub use types::OutcomeKind;
pub use types::OutcomeResult;
pub use types::Provenance;
pub use types::RangePlanDelta;
pub use types::SelectedPeer;
pub use types::Selection;
pub use types::TransferOutcome;

Modules§

config
SelectorConfig — wiring only, never behavior (SPEC.md §1.5, §5.6).
engine
PeerSelector — the public engine (SPEC.md §5): the closed select → record_outcome → rebalance decision + learning loop over the registry (§2), the measured-only quality model (§3), and the autonomous scorer (§4).
observe
Read-only observability snapshots (SPEC.md §5.7) — the agent-friendly / machine-consumable surface.
pool_event
PoolEvent / PoolRemovalReason — the topology/churn feed, byte-compatible with dig_gossip::PoolEvent (SPEC.md §5.4, §7.2).
quality
PeerQuality — the per-peer, measured-only capacity/RTT/reliability model (SPEC.md §3).
registry
The dynamic peer registry: peer_id -> PeerEntry, fed by churn + DHT candidates, bounded with lowest-value eviction (SPEC.md §2).
scoring
The autonomous scoring function (SPEC.md §4): learned saturation per class, an adaptive relayed penalty, volatility-driven decay (in crate::quality), the min-P99 + anti-thundering-herd objective, and the ranked-subset output — satisfying invariants A–G (SPEC §4.4).
types
The frozen public value types of the selector API (SPEC.md §5.1, §5.2, §5.5).

Structs§

CandidateAddr
The candidate/content types re-used from dig-dht (SPEC §7.1): what is fetched + how a provider is addressed. One candidate address for a provider: { host, port, kind } (L7 dig.getPeers §7). The finder dials these (most-direct-first) via dig_nat::connect to reach the provider.
PeerId
The transport-verified peer identity (peer_id = SHA-256(TLS SPKI DER)), re-used from dig-nat. A peer’s stable network identity: the 32-byte SHA-256 of its TLS SPKI DER.
ProviderRecord
The candidate/content types re-used from dig-dht (SPEC §7.1): what is fetched + how a provider is addressed. The DHT’s stored value: peer provider_peer_id holds the content whose key is content_key, reachable at addresses, until expires_at.

Enums§

AddressKind
The candidate/content types re-used from dig-dht (SPEC §7.1): what is fetched + how a provider is addressed. How a candidate address was learned — the L7 dig.getPeers addresses[].kind tokens (§7). The lowercase serde spelling is the frozen wire form; the ordering is most-direct-first (a dialer picks the lowest-rank dialable candidate).
ContentId
The candidate/content types re-used from dig-dht (SPEC §7.1): what is fetched + how a provider is addressed. A DIG content identifier at store / root(capsule) / resource granularity — the key a provider record is stored under and a lookup asks for.
TraversalKind
The dig-nat connection-class ladder the selector reads observationally (SPEC §7.3). Which traversal technique produced a result — used to order methods, tag failures, and report (observability) which method actually succeeded WITHOUT the caller caring.