# ADDRESSING.md — Addresses are Multiaddrs, and addresses route themselves
## Premise
In bytes-and-brains, an **address is a multiaddr** — a sequence of typed
protocol segments. The multiaddr fully describes the delivery path, so the
receiver does not consult any per-message subscription table, opset routing
map, or `recv_sites`-style HashMap. The address itself is the routing.
Sender:
1. Builds the destination multiaddr by walking the graph: the producer's
compiler pass knows which Recv-op site or which Component-op the value
needs to land at.
2. Packs each typed input into a `SlotFill` with its per-slot multiaddr
suffix.
3. Resolves the destination `PeerId` to its ordered address list via
the framework's `AddressBook` and packs that list into
`WireEnvelope.dest_peer_addresses`. Sends one envelope per
`(producer_op, dest_peer_addresses_first)` group.
4. Snapshots its own local-address bag (`ctx.local_addresses()` at
`bb-runtime/src/runtime.rs:269`) and stamps it onto
`WireEnvelope.src_peer_addresses` (`bb-ops/src/network/wire/mod.rs:196-220`)
so the receiver learns every interface the sender currently binds.
Receiver:
1. Parses each fill's `dest_suffix`.
2. The suffix's trailing segments tell the receiver exactly which slot
or component-op to deliver to.
3. Dispatches directly — no subscription lookup.
4. Merges `envelope.src_peer_addresses` and the transport-observed
`IngressEvent::EnvelopeFrom.src_observed_address` into the
receiver's AddressBook entry for the source peer
(`bb-runtime/src/engine/poll.rs:1005-1062`). Claimed addresses land
first; observed wins for the NAT-translated case the sender's
snapshot cannot know.
The DAG / IR / DSL still key destinations by `PeerId`. The wire
syscall is the resolution boundary: it consults the
[`AddressBook`](#address-book-semantics) at dispatch time, packs the
resolved list into the envelope, and surfaces
[`EngineStep::PeerResolveFailed`](#peer-resolution-failure) on miss.
## Protocol enum
The framework carries the per-slot `dest_suffix` in each `SlotFill`
for intra-node routing AND the resolved ordered address list in
`WireEnvelope.dest_peer_addresses` for transport selection. Peer
resolution (`PeerId → Vec<Address>`) happens inside the wire syscall
via the `AddressBook`; the host's transport adapter just picks one
of the supplied addresses based on its networking capabilities.
```rust
pub enum Protocol {
P2p(PeerId), // varint 421 — libp2p-standard /p2p/<multihash>
Site(NodeSiteId), // varint 0xE2 — data-plane slot fill target
Component(ComponentRef),// varint 0xE3 — control-plane component
Op(String), // varint 0xE4 — control-plane op discriminator
}
```
`NodeSiteId` is globally unique within a Node (allocated from a
global `AtomicU64` counter), so `/site/<id>` uniquely identifies a
data-plane slot. `Component + Op` together identify a control-plane
dispatch target.
### Why so narrow?
The engine's routing dispatch (`Engine::deliver_fill` in
`src/engine/poll.rs`) ONLY matches on `addr.site_id()` and
`addr.component_ref() + addr.op_name()`. Every other multiaddr
segment was historically supported for libp2p round-trip symmetry
(Ip4/Ip6/Tcp/Udp), logs (Graph), or test fixtures (Memory) — none
of which the framework ever read for routing. They were
maintenance burden with zero internal benefit.
**Host adapter responsibility.** When a host (libp2p, in-process
sim, anything else) ships an outbound envelope, it reads
`dest_peer_addresses` (a `repeated bytes` already resolved by the
wire syscall via the `AddressBook`) and picks one entry based on
its networking capabilities (IPv4 reachability, QUIC support,
relay preference, etc.). The adapter then constructs whatever
transport-level bytes it needs and ships the envelope. On receipt,
the adapter passes the envelope into the framework along with the
source `PeerId` (via `IngressEvent::EnvelopeFrom { src_peer,
envelope }`) — the framework dispatches purely by `dest_suffix` on
each fill, ignoring `dest_peer_addresses` on the inbound side.
## Binary encoding
`Address` encodes as `varint(code) || payload`. Protocol codes are
unsigned LEB128 varints — matches libp2p's multiaddr wire format.
- `/p2p/`: payload is `varint(len) || multihash bytes`, identical
to libp2p byte-for-byte. The multihash itself carries an
algorithm code (identity, sha2-256, sha3, blake2b, …) so
different overlay protocols pick their own digest function
without coordination.
- `/site/`: payload is a fixed 8-byte big-endian `NodeSiteId`.
- `/component/`: payload is a fixed 4-byte big-endian
`ComponentRef`.
- `/op/`: payload is `varint(len) || utf8 bytes`.
Concatenated segments form the full byte sequence. Empty Address
= empty buffer.
`Address::from_bytes` rejects any varint code outside the
framework's set with `AddressError::UnknownCode`. Standard libp2p
transport codes (`ip4 = 4`, `tcp = 6`, etc.) fail to parse —
they're host-adapter concerns that should never reach the
framework. The `/p2p/` code (421) is recognized so a multiaddr
suffix produced by libp2p's `Multiaddr` round-trips through our
codec at the wire level.
## String form
`/protocol/value/protocol/value/...` — segments separated by `/`,
each protocol's value parsed per its type. The `/p2p/<peerid>`
form uses base58btc-encoded multihash bytes, matching libp2p's
`Qm…` / `12D3…` peer-id strings exactly:
```
/p2p/12D3KooWBh.../site/17
→ data-plane fill on peer 12D3KooWBh..., slot 17
/p2p/12D3KooWBh.../component/7/op/FindNode
→ control-plane component-7's "FindNode" op on peer 12D3KooWBh...
```
`Address::from_str` parses the string form; `Display` round-trips
back. Unknown protocols (including `ip4`, `tcp`, `memory`, etc.)
return `AddressError::InvalidValue`.
## Wire envelope
The wire envelope ships one **resolved destination address list**,
the sender's **claimed local-address bag**, and one or more
`SlotFill`s, batched per the compiler's `analyze_wire_edges` pass.
```proto
message WireEnvelope {
// Ordered destination address list — the framework's wire
// syscall populates this from the AddressBook at dispatch time.
// The host's transport adapter picks one based on its
// networking capabilities. Each entry is `Address::to_bytes()`.
repeated bytes dest_peer_addresses = 1;
repeated SlotFill fills = 2;
WireCorrelation correlation = 3;
// ... deadline + RTT piggyback + src_peer_bytes + schema_version ...
// Sender-claimed local-address bag — snapshot of the sender's
// AddressBook entry for its own PeerId at envelope-mint time.
// Receiver merges into its own AddressBook entry for the sender
// (see Receiver-side address ingest). Each entry is
// `Address::to_bytes()`; bounded at decode time by
// `EnvelopeCaps.max_src_peer_addresses` +
// `max_src_peer_address_bytes`.
repeated bytes src_peer_addresses = 8;
}
message SlotFill {
bytes dest_suffix = 1; // per-slot multiaddr suffix (intra-node)
bytes payload = 2;
bool trigger_only = 3;
}
```
Peer routing and intra-node slot routing are decoupled:
- **`dest_peer_addresses`** is the resolved snapshot of
`AddressBook::lookup(peer)` at dispatch time — an ordered slice
by peer-stated preference. The host's transport adapter picks
one of the entries based on its networking capabilities (IPv4
reachability, QUIC support, relay preference, etc.). The DAG /
IR / DSL still address destinations by `PeerId` — only the
envelope-on-the-wire carries the resolved list.
- **`src_peer_addresses`** (proto field 8, `proto/bb_core.proto:135`)
is the snapshot of the sender's `local_addresses()` at envelope-mint
time. The wire syscall stamps it on every Send
(`bb-ops/src/network/wire/mod.rs:196-220`); receivers feed it into
the AddressBook merge described below. Empty stamps an empty
`repeated bytes` so the receiver leaves its existing entry alone.
- **`dest_suffix`** is the per-slot multiaddr suffix the receiver
parses to dispatch — `/site/<NodeSiteId>` for data-plane fills or
`/component/<cref>/op/<name>` for control-plane fills.
`dest_peer_addresses` routes to a peer; the suffix routes within
the receiver; `src_peer_addresses` keeps both peers' AddressBooks
in sync without an out-of-band exchange.
`EnvelopeCaps.max_src_peer_addresses` (default 8;
`bb-runtime/src/envelope.rs:44`) and
`max_src_peer_address_bytes` (default 256; `bb-runtime/src/envelope.rs:48`)
cap the receiver's pre-allocation so an adversarial sender cannot
balloon the AddressBook through inflated claims. `EnvelopeCodec::decode_capped`
checks both caps before any prost allocation
(`bb-runtime/src/envelope.rs:236-246`).
## Receiver dispatch
For each fill in an inbound envelope, the engine parses `dest_suffix`
and routes by the trailing-segment shape:
| ends in `/site/<id>` | Data-plane: charge `fill.payload.len()` against `Engine::ingress_byte_budget`; resolve typed `SlotValue` via `decode_typed_fill` (`bb-runtime/src/engine/poll.rs:996-1083`); write to slot at `NodeSiteId`; push consumers. Backend-bound slots route through `Backend::materialize_from_wire`; non-backend slots run the global `wire_decoder_registry`. |
| ends in `/component/<cref>/op/<name>` | Control-plane: call `components[cref].dispatch_atomic(name, [(payload, correlation)], ctx)` |
| anything else | Silent drop |
For data-plane fills with `trigger_only = true`, the receiver writes a
`TriggerValue` instead of decoding the payload (which is empty).
**Self-addressed wire ops with multi-target Nodes.** When a peer
hosts more than one target (e.g. `&["Client", "Server"]`), the
wire syscall continues to address destinations by `PeerId` and the
receiver dispatches by `dest_suffix` exactly as in the single-target
case. The receiver's site name → `NodeSiteId` table is **global
across every installed target's graph** (`Engine::install_graph`
populates a single `site_names` map for every installed graph), so
a `wire.Send` from `Client`'s partition addressed at
`self.peer_id()` lands at the correct `wire.Recv` site even when
that site lives inside `Server`'s partition graph. The site_id is
unique per Node, not per target, so the partition the suffix
resolves into is determined entirely by which graph owns the
`NodeSiteId` — no extra target-awareness is required in the
envelope or the dispatch table. The same holds for control-plane
`/component/<cref>/op/<name>` fills: the deduped `ComponentRef` is
shared across every target sharing the slot, so a control-plane
fill from `Client`'s partition reaches the same component instance
the `Server` partition would dispatch to.
**The receiver does not allocate envelope-receive storage outside
the fallibility line.** `fill.payload` arrives from prost
already framework-owned (envelope decode allocated it under the
`EnvelopeCaps::max_total_bytes` cap, `bb-runtime/src/envelope.rs:197-207`).
Per-fill failures (`AllocationFailed`, `BudgetExceeded`,
`BackendMaterializeFailed`, decode failures) drop the offending
fill and continue iterating sibling fills — partial-delivery
semantics per [WIRE.md §5.4](WIRE.md#per-fill-failure-modes).
The receiver-side address-book hint is best-effort under the
byte budget: an `AddressBook` add that returns
`AddressBookError::AllocationFailed` is swallowed so the
envelope still routes; the routing decision never depends on
the address-book write succeeding.
## Local address bag
A Node binds to `Vec<Address>` at install time, not a single
`Address`. Real deployments bind to several interfaces at once: an
IPv4 public address, an IPv6 public address, a tailscale virtual,
an HTTP endpoint, a libp2p relay. Every interface is registered
against the Node's own `PeerId` in the `AddressBook` and rides
unmodified on every outbound envelope.
`bb::install(peer_id, addresses, model, targets, config)` at
`src/install.rs:235-241` takes the bag verbatim. An empty vec
skips self-registration so the Node renders all outbound
identity-protocol ops as "no addresses" failures at the protocol
level (the golden way: no synthesized `/p2p/<PeerId>` fallback).
The `addresses` bag is shared across every entry in the
`targets: &[&str]` slice — every target the install path registers
sees the same `local_addresses()` view, since the bag is owned
once by the engine's `AddressBook` keyed on the Node's
`self_peer`.
Three accessor methods sit on `Node` for runtime mutation
(`bb-runtime/src/node/mod.rs:625-660`):
```rust
pub fn local_addresses(&self) -> &[Address];
pub fn add_local_address(&mut self, addr: Address) -> Result<(), AddressBookError>;
pub fn forget_local_address(&mut self, addr: &Address) -> Result<(), AddressBookError>;
```
Each is a one-line wrapper around the AddressBook entry keyed by
`self.engine.self_peer`. `peer_address()` returns the first entry
or `Address::empty()` for source compatibility with single-address
call sites (`bb-runtime/src/node/mod.rs:611-619`).
Every dispatch context exposes `ctx.local_addresses()`
(`bb-runtime/src/runtime.rs:269-274`): every Contract impl that
populates "here are my addresses" reads through this accessor so
mutations via `node.add_local_address` propagate without rebinding.
## Receiver-side address ingest
Phase 1 of `Engine::poll` reads both sender-claimed and
transport-observed addresses for every inbound envelope. The two
sources land via separate paths into the same merge code at
`bb-runtime/src/engine/poll.rs:1005-1062`:
- **Claimed (sender)** — `envelope.src_peer_addresses` decodes into
a `Vec<Address>`. Empty list is a no-op (sender did not advertise).
Skip-on-unchanged guard: slice equality against the existing
AddressBook entry elides the rewrite, capping cost at one
AddressBook write per real change.
(`merge_src_peer_addresses` at lines 1013-1039.)
- **Observed (transport)** — `IngressEvent::EnvelopeFrom.src_observed_address`
carries the dialer endpoint the adapter actually saw. `None` means
the adapter cannot surface a reflexive address. Containment check
against the existing entry skips the rewrite; missing entry
bootstraps a fresh one via `add_peer`. (`merge_src_observed_address`
at lines 1049-1062.)
Claimed addresses merge first so the entry exists when the observed
merge tries to append; observed wins for the NAT-translated case
because the sender's snapshot cannot know its post-NAT endpoint.
The in-process router defaults `src_observed_address` to
`Some(Address::empty().p2p(src_peer))` via
`IngressEvent::from_in_process` (`bb-runtime/src/ingress.rs:143-152`)
so the test surface exercises the merge path real transports
populate.
## Address book semantics
The framework's `AddressBook` is the single source of truth for
every `PeerId → Vec<Address>` mapping. Overlay protocols, transport
adapters, and the application share it; no Component duplicates
address bytes in its own state.
### Ref-counted multi-address entries
Each entry is `(addresses: Vec<Address>, ref_count: u64)`:
- **`addresses`** is ordered by insertion. Earlier entries are
higher-preference; new addresses appended to a known peer
preserve the existing order with dedupe (the first caller's
preference wins).
- **`ref_count`** tracks how many independent owners (overlay
protocols, transport adapters, the application) hold a grip on
the peer. An entry survives until its last reference drops.
### Mutations
| `add_peer(peer, Vec<Address>)` | `+1` (new entry starts at 1) | merge-append with dedupe; preserves order |
| `drop_peer(peer)` | `-1`; entry removed at 0 | (entry removal also drops all bound addresses) |
| `register_address(peer, addr)` | unchanged | append `addr` with dedupe |
| `forget_address(peer, addr)` | unchanged | prune `addr`; entry stays even if list becomes empty |
`add_peer` rejects an empty `Vec<Address>` with
`AddressBookError::EmptyAddressList`. `drop_peer` /
`register_address` / `forget_address` error with
`AddressBookError::UnknownPeer` when the peer has no entry.
`AddressBook::lookup(peer)` returns `Option<&[Address]>`. It yields
`None` for both an unknown peer AND a peer whose address list is
empty (e.g., all addresses pruned via `forget_address` and nothing
re-added). Both are treated as "can't route" by the wire syscall.
## DAG-mutable address book — `address_book` syscall ops
The `ai.bytesandbrains.address_book` opset exposes two primitives
recorded graphs use to seed the `AddressBook` from the data plane.
Both are niche but load-bearing for discovery protocols (mDNS,
gossip-overlay address propagation, relay-discovered peer
advertisement): they let those protocols compile into a Graph that
announces and resolves peers without out-of-band host calls.
| `Insert(peer, address) → ()` | (runtime-internal; recorded directly) | new peer → `add_peer(peer, vec![addr])`; known peer → `register_address(peer, addr)` | empty input, `Full` |
| `InsertMany(peer, addresses) → ()` | `bb_dsl::syscalls::address_book_insert_many(g, peer, addrs)` (`bb-dsl/src/syscalls.rs:55-66`) | new peer → `add_peer(peer, addrs)`; known peer → one `register_address` per address | empty input, `Full` |
| `Lookup(peer) → addresses: Vec<Address>` | `bb_dsl::syscalls::address_book_lookup(g, peer)` (`bb-dsl/src/syscalls.rs:72-83`) | full ordered slice via `AddressBook::lookup` | unknown peer, empty list |
`Insert` (`bb-ops/src/syscalls/peers/insert.rs`) is the single-address
path the receiver-side merge inside the engine reaches for.
`InsertMany` (`bb-ops/src/syscalls/peers/insert_many.rs`) is the
batched form discovery protocols use to record an entire peer bag
in one NodeProto. `Lookup` returns the full ordered slice on the new
`TYPE_ADDRESS_VEC` carrier (`bb-ir/src/types/builtins.rs:311-318`);
callers that need a single entry pick one explicitly.
Address forwarding scenarios (relay, mDNS, DHT address advertisement)
are graph-level concerns: a forwarding Component records a graph
that consumes a `(PeerId, Vec<Address>)` payload + calls
`InsertMany`. Address bytes ride in the payload, not in the envelope
header.
### Carriers
`bb-runtime/src/syscall/values.rs:67-68` defines `AddressVecValue` as
the wire-eligible carrier for the ordered bag. The lattice node
`TYPE_ADDRESS_VEC` (`bb-ir/src/types/builtins.rs:311-318`) registers
under `ai.bytesandbrains.address_vec` with wire-hash 0x0303 so a
single `Lookup` output and an `InsertMany` input share one decoder
across the wire.
## Peer resolution failure
`wire::Send` cannot resolve its destination in three cases:
- the `peer` input didn't carry a parseable `PeerId`;
- the `PeerId` is unknown to the `AddressBook`;
- the `PeerId`'s entry exists but its address list is empty.
In each case the syscall produces NO envelope, pushes a record onto
the framework's `pending_peer_resolve_failures`, and publishes
`InfraEvent::PeerResolveFailure` on the bus. The engine's Phase 8
drains the queue and surfaces each entry as
`EngineStep::PeerResolveFailed { peer, op_ref, exec_id }`.
This is a first-class lifecycle event in the same family as
`PeerBlocked` / `PeerDown` / `PeerUp` from Production-Readiness
Stage 4 — not an `OpFailed`. Telemetry / dashboards can surface
resolution failures alongside other peer-lifecycle signals.
## Address construction at install time
Two-step resolution per producer Send NodeProto:
1. **Compiler** (`analyze_wire_edges`, [COMPILER.md §9](COMPILER.md))
— for each cross-Node edge, stamps the destination multiaddr's
recv-side symbolic value name as
`metadata_props["ai.bytesandbrains.dest_site_name.<input>"] = "<recv_output_name>"`.
The name is symbolic because runtime `NodeSiteId`s aren't allocated
until install.
2. **Node install** — after all `ModelProto` graphs are
installed, walks each Send NodeProto, looks up every
`dest_site_name.<input>` value against the global `site_names`
map (spanning every installed graph), and rewrites it as a
canonical `dest_suffix.<input>` `AttributeProto` carrying
`Address::empty().site(NodeSiteId).to_bytes()`.
At dispatch time the wire syscall's `collect_fills` reads each
fill's suffix from the resolved `dest_suffix.<input>` attribute
via `ctx.current_node_attributes`. A `<name>_suffix` companion
input is supported as a fallback for sim harnesses that construct
envelopes manually outside the compilation pipeline.
## What this replaces
| `Address(Vec<u8>)` opaque bytes | typed `Protocol` segment sequence |
| `Engine.recv_sites: HashMap<String, Vec<(OpRef, NodeSiteId)>>` | parse `Address::site_id()` from each fill |
| `Engine.event_subscriptions: HashMap<String, Vec<OpRef>>` | `HashMap<String, Vec<NodeSiteId>>` — Phase 3 writes a `TriggerValue` to each subscribed site and pushes consumers (same dispatch path as wire fills) |
| `Engine.routing_table: RoutingTable` for `(opset, version) → DataPlane/Control` | parse `dest_suffix` segments |
| `WireEnvelope.opset / message_type / peer_id / hash` | encoded as multiaddr segments |
| `WireEnvelope.dest_peer: uint64` (single PeerId on the wire) | `WireEnvelope.dest_peer_addresses: repeated bytes` — resolved address list; transport picks one |
| `AddressBook: PeerId → Address` (single address per peer) | `AddressBook: PeerId → (Vec<Address>, ref_count)` — ordered list + ref-counting |
| `PeerAddressBook: PeerId → (identity, Address)` (duplicated address) | `PeerAddressBook: PeerId → identity` only — addresses live solely in `AddressBook` |
| `ModelProto.recv_index` / `RecvSiteRef` | gone — compiler stamps `dest_site_name.<input>` on each producer Send NodeProto; Node install resolves to `dest_suffix.<input>` Address bytes via the global `site_names` map |
| `Engine.wire_inbox` (fallback for unaddressed payloads) | every fill is addressed; malformed addresses drop silently |
| `Engine.framework.address_book` / `peer_address_book` were transient-only | snapshot/restore via `FrameworkSnapshot.address_book` + `peer_address_book` — peer registries persist across restarts |
## Code map
| `Address`, `Protocol`, `AddressBook`, `AddressBookError` | `bb-runtime/src/framework/address_book.rs` |
| `PeerAddressBook` (identity only) | `bb-runtime/src/framework/peer_address_book.rs` |
| Wire envelope proto (`src_peer_addresses` at field 8) | `proto/bb_core.proto:135` |
| Envelope encode/decode + `EnvelopeCaps` (claim caps) | `bb-runtime/src/envelope.rs:42-48,236-246` |
| `AddressVecValue` typed carrier | `bb-runtime/src/syscall/values.rs:59-65,101` |
| `TYPE_ADDRESS_VEC` lattice node | `bb-ir/src/types/builtins.rs:306-318,369,409` |
| Outbound `wire.Send` (stamps `src_peer_addresses`) | `bb-ops/src/network/wire/mod.rs:196-220` |
| `address_book/` syscall ops (`Insert`, `InsertMany`, `Lookup`) | `bb-ops/src/syscalls/peers/{insert,insert_many,lookup}.rs` |
| DSL helpers (`address_book_insert_many`, `address_book_lookup`) | `bb-dsl/src/syscalls.rs:50-83` |
| `EngineStep::PeerResolveFailed` | `bb-runtime/src/engine/step.rs` |
| `InfraEvent::PeerResolveFailure` | `bb-runtime/src/bus.rs` |
| Inbound parse + dispatch + ingest merge | `bb-runtime/src/engine/poll.rs:473-514,1005-1062` |
| `IngressEvent::EnvelopeFrom.src_observed_address` | `bb-runtime/src/ingress.rs:47-60,143-152` |
| `Node::install(peer_id, Vec<Address>, ...)` entry | `src/install.rs:236-242` |
| `Node` local-address accessors | `bb-runtime/src/node/mod.rs:611-660` |
| `ctx.local_addresses()` | `bb-runtime/src/runtime.rs:262-274` |
| `GlobalRegistry::Announce` / `Handshake` carry full bags | `bb-ops/src/protocols/global_registry/mod.rs:56-72,162-186,477-543` |
| Tests: address round-trips + ref-counting | `bb-runtime/src/framework/address_book_tests.rs` |
| Tests: `address_book` syscall ops | `bb-ops/src/syscalls/peers/{insert,insert_many,lookup}_tests.rs` |
| Tests: envelope shapes + `src_peer_addresses` caps | `bb-runtime/src/envelope_tests.rs` |
| Tests: ingress merge (claimed + observed) | `bb-runtime/src/engine/poll_src_peer_addresses_tests.rs`, `bb-runtime/src/engine/poll_observed_address_tests.rs` |
| Tests: end-to-end multi-address delivery | `tests/local_multi_address_e2e.rs`, `tests/wire_envelope_src_addresses.rs` |