# Architecture
This page explains how the core Rust library is organized, how telemetry moves through it, and why the data structures
look the way they do. It assumes no prior knowledge of the codebase.
## Goals and constraints
- **Runtime schema**: endpoints, message types, and payload layouts live in a runtime registry that can be seeded,
queried, exported, and synced between nodes.
- **Compact on-the-wire representation**: a small header, endpoint bitmaps, and optional compression minimize bandwidth.
- **Embedded-friendly**: no_std support, bounded queues, and stack/heap tradeoffs.
- **Multi-language**: Rust, C, and Python expose the same runtime schema and packet APIs.
- **Deterministic behavior**: validation and dedupe rules are consistent across languages.
## Module map (what lives where)
- [src/config.rs](https://github.com/Rylan-Meilutis/sedsnet/blob/main/src/config.rs): packaged
configuration defaults, runtime tuning state, and the runtime schema registry for `DataType` and
`DataEndpoint` metadata.
- [src/lib.rs](https://github.com/Rylan-Meilutis/sedsnet/blob/main/src/lib.rs): schema metadata
(`MessageMeta`, `MessageElement`, `MessageDataType`, `MessageClass`).
- [src/packet.rs](https://github.com/Rylan-Meilutis/sedsnet/blob/main/src/packet.rs): `Packet`
validation, formatting, packet IDs, and migration-safe wire-contract state.
- [src/small_payload.rs](https://github.com/Rylan-Meilutis/sedsnet/blob/main/src/small_payload.rs):
inline-optimized payload storage (`SmallPayload`).
- [src/wire_format.rs](https://github.com/Rylan-Meilutis/sedsnet/blob/main/src/wire_format.rs): compact
wire format, ULEB128 helpers, envelope peek, and packet IDs from wire.
- [src/router.rs](https://github.com/Rylan-Meilutis/sedsnet/blob/main/src/router.rs): router core,
queues, endpoint handlers, side-based routing, discovery, and compact side transport.
- [src/relay.rs](https://github.com/Rylan-Meilutis/sedsnet/blob/main/src/relay.rs): schema-agnostic
fanout relay between sides.
- [src/queue.rs](https://github.com/Rylan-Meilutis/sedsnet/blob/main/src/queue.rs): bounded deque used
by router and relay.
- [src/c_api.rs](https://github.com/Rylan-Meilutis/sedsnet/blob/main/src/c_api.rs) and
[src/python_api.rs](https://github.com/Rylan-Meilutis/sedsnet/blob/main/src/python_api.rs): FFI
bindings (C ABI and pyo3).
## Runtime schema and metadata pipeline
The registry starts with built-in internal endpoints/types for telemetry errors, reliable control,
discovery, and time sync. User endpoints and data types are added at runtime through direct APIs,
optional JSON seeding, handler registration, or discovery schema sync.
Core schema types:
- `DataEndpoint`: transparent runtime ID (`u32`) for logical destinations.
- `DataType`: transparent runtime ID (`u32`) for message types.
- `MessageMeta`: per-`DataType` metadata (name, element layout, allowed endpoints).
- `MessageElement`: `Static(count, MessageDataType, MessageClass)` or `Dynamic(MessageDataType, MessageClass)`.
- `MessageDataType`: primitive element type (Float32, UInt16, String, Binary, etc.).
- `MessageClass`: logical class (`Data`, `Warning`, `Error`).
Why these structures exist:
- **MessageMeta is registry-backed** so it can change as peers add or announce schema over time.
- **String lookup APIs** (`DataEndpoint::named`, `DataType::named`) keep application code readable while the wire format
remains numeric.
- **MessageElement separates shape from data type** so the same primitive type can be used for static and dynamic
payloads.
- **MessageClass is tied to element layout** so formatting and error generation know the intent of the message.
- **Discovery schema sync** exports endpoint/type definitions, merges compatible remote schema, and resolves conflicts
deterministically.
## Packet (the core runtime type)
`Packet` is the validated, shareable container used by the router and FFI layers. It contains:
- `ty: DataType` (logical message type).
- `data_size: usize` (cached payload length; must match `payload.len()`).
- `sender: Arc<str>` (logical sender identifier).
- `endpoints: Arc<[DataEndpoint]>` (destinations; must be non-empty).
- `timestamp: u64` (ms; treated as uptime below 1e12, or epoch ms otherwise).
- `payload: StandardSmallPayload` (inline optimized; see below).
- `wire_shape: Option<MessageElement>` when the frame carried inline decode metadata.
- `wire_target_senders: Arc<[u64]>` when the frame carried a frozen destination-holder set.
Validation rules enforced by `Packet::new` and `Packet::validate`:
- Endpoints list must be non-empty.
- For static layouts, payload length must be exactly `count * data_type_size`.
- For dynamic layouts:
- Numeric/bool types: payload length must be a multiple of element width.
- String types: trailing NULs are ignored and remaining bytes must be valid UTF-8.
- Binary types: no UTF-8 requirement (raw bytes).
Packet IDs are **not** packed. `Packet::packet_id` hashes:
- compact source address derived from the sender/discovery mapping
- message name (`DataType` as string)
- endpoint names (in order)
- timestamp + data_size (little-endian)
- payload bytes
This produces the same ID across different links, which enables de-duplication.
### SmallPayload and inline storage
`SmallPayload<INLINE>` stores short payloads directly on the stack and spills to `Arc<[u8]>` when larger. The inline
size is controlled by `MAX_STACK_PAYLOAD` via `define_stack_payload!`, and the default inline capacity is 64 bytes.
That value is a compiled capacity because it affects the stack payload type layout; active static string/binary sizing,
compression threshold, retry counts, reliable queue caps, node identity, memory budgets, and time-sync roles are runtime
configuration.
Why this matters:
- Small telemetry values avoid heap allocation and `Arc` traffic.
- Larger payloads still have cheap clone semantics because they are `Arc`-backed.
## Serialization and wire format
The compact v2 wire format is implemented in
src/wire_format.rs ([source](https://github.com/Rylan-Meilutis/sedsnet/blob/main/src/wire_format.rs)).
A packet is encoded as a compact v2 frame with schema-derived endpoints or an explicit endpoint bitmap, optional wire contract, optional reliable header, payload bytes, and a CRC32 trailer. See [Technical-Wire-Format](Technical-Wire-Format) for the exact field order.
Design choices:
- **ULEB128 varints** minimize size for small values.
- **Schema-derived endpoints** omit endpoint bytes when a packet uses the data type's default endpoint set.
- **Endpoint bitmap** represents custom endpoint sets; width is fixed from `MAX_VALUE_DATA_ENDPOINT`, not from the current runtime registry contents.
- **Wire contract** carries inline payload shape and frozen destination-holder sender hashes so in-flight packets stay decodable and correctly targeted across topology/schema churn.
- **Sender/payload compression** is applied independently only when it reduces the wire size.
- **CRC32 trailer** validates the entire frame before normal decode.
`packet_id_from_wire` parses only as much as needed to compute the same packet ID as `Packet::packet_id`. It
always hashes **decompressed** sender/payload bytes, so dedupe works across compressed and uncompressed links.
`TelemetryEnvelope` is a header-only view that can be obtained by `peek_envelope` without copying the payload.
## Queues and bounded memory
Router and relay queue storage is built on `BoundedDeque<T>`:
- Byte-budgeted: each item reports a `byte_cost` via the `ByteCost` trait.
- Hard element cap: capacity never exceeds `max_elems` (computed from `size_of::<T>()`).
- Eviction policy: pop from the front until new item fits in byte budget.
- Growth policy: multiplicative growth controlled by `QUEUE_GROW_STEP`.
The public `MAX_QUEUE_BUDGET` is shared dynamically across router/relay RX queues, TX queues,
recent-ID caches, reliable replay/out-of-order buffers, discovery topology state, and runtime
schema registry memory. Recent-ID caches preallocate their final storage at construction and
reserve that amount from the shared budget immediately. This design keeps memory use bounded and
avoids unbounded `VecDeque` growth in embedded targets while letting the active part of the system
use available queue budget when other internal queues are quiet.
## Router architecture
The router is the main API for logging, receiving, dispatching, and (optionally) relaying.
Key structures:
- `RouterConfig`: holds local `EndpointHandler` definitions in an `Arc<[EndpointHandler]>`.
- `EndpointHandler`: packet or packed handler for a specific `DataEndpoint`.
- `RouterSideId`: identifies a named side (UART/CAN/RADIO/etc.).
- `RouterItem`: either `Packet` or packed bytes.
Receive flow:
1) RX bytes or packet are processed immediately or queued (`rx_*` vs `rx_*_queue`).
2) Reliable side headers are handled first when the ingress side has reliable delivery enabled.
3) Packet ID is computed. For packed bytes, the router tries `packet_id_from_wire` and falls back to hashing raw
bytes if needed.
4) If the packet ID is already in the recent cache, it is dropped.
5) Local endpoint handlers are invoked.
6) The router forwards the packet according to the current route rules, discovery/path-selection state, and any frozen destination-holder contract attached to the packet.
Forwarding is driven by:
- the active side ingress/egress policy
- route overrides and typed route overrides
- discovery-learned endpoint reachability
- link-local side scope
- route-selection policy (`Fanout`, `Weighted`, `Failover`)
Transmit flow:
- `log*` builds a new `Packet` from typed data, validates sizes, and packs it.
- `tx*` sends a pre-built `Packet` (validated) or packed bytes.
- Queue variants (`*_queue`) defer the work until `process_*_queue`.
Error handling:
- Local handler failures are retried using the active runtime `max_handler_retries` value.
- If a handler ultimately fails, the packet ID is removed from the dedupe cache so a resend can be processed later.
- The router emits a `TelemetryError` packet for local handlers when possible.
## Relay architecture
`Relay` is schema-agnostic and purely forwards packets between named sides (UART/CAN/RADIO/etc.).
- Each side registers a TX handler for packed bytes or full packets.
- Each side can opt out of reliable sequencing/ACKs (useful for TCP links).
- The relay maintains RX/TX queues and a recent-ID cache like the router.
- Fanout clones the `Arc` for payload sharing; it does not decode unless needed for a handler.
- Dedupe ignores the source side so duplicates from different links are dropped.
## Thread safety and time
Routers and relays use an internal `RouterMutex` so that most APIs can take `&self` while still protecting shared state.
The clock source is injected via the `Clock` trait so embedded builds can supply their own timing source.
## End-to-end lifecycle (summary)
```
Producer Router/Relay Consumer
-------- ------------- --------
log()/tx() -> validate -> pack -> tx(bytes, link) -> transport -> rx_packed()
rx(bytes) -> unpack -> dedupe -> handlers -> (optional) relay forward
```
If you need build-time or schema details, see [Build-and-Configure](Build-and-Configure) and
[Technical-Telemetry-Schema](Technical-Telemetry-Schema).