# corpus-ipc
[](https://github.com/Limen-Neural/corpus-ipc/actions/workflows/ci.yml)
[](https://codecov.io/gh/Limen-Neural/corpus-ipc)
[](https://crates.io/crates/corpus-ipc)
[](https://docs.rs/corpus-ipc)
[](https://app.codacy.com/gh/Limen-Neural/corpus-ipc/dashboard?utm_source=gh&utm_medium=referral&utm_content=&utm_campaign=Badge_grade)
[](https://qlty.sh/gh/Limen-Neural/projects/corpus-ipc)
[](https://coderabbit.ai)
[](#license)
Spike and neuromodulator wire protocol and IPC transports for Limen-Neural SNN runtimes.
`corpus-ipc` is the schema and transport layer for those runtimes. It provides backend abstractions for local/native execution and optional ZeroMQ IPC, plus canonical wire message models used across services.
## Features
- `IpcBackend` trait for backend-agnostic signal processing (deprecated alias: `RuntimeBackend`)
- `RustBackend` reference backend (always available)
- `ZmqIpcBackend` backend via ZMQ SUB socket (feature `zmq`; deprecated alias: `ZmqRuntimeBackend`)
- Optional `corpus_ipc_server` REST binary (feature `server`; not compiled for library consumers)
- Canonical protocol models:
- `IpcMessage`
- `SpikeBatch`, `SpikeEvent` (IPC wire types; alias `IpcSpikeBatch`)
- `EmbeddingBatch`
- `StimulusBatch` for typed, variable-width runtime stimulus ingress
- `GradientBatch`, `GradientUpdate`
- `TraceBatch`, `TraceData` (IPC wire types; alias `IpcTraceBatch`)
- `ConfigPayload`, `ConfigValue`, `BatchMetadata`
- `HybridFlowBackend` trait for message-oriented hybrid transports
- `NeuromodulatorSnapshot` for typed neuromodulator ingress/readout payloads
- `WireCompatibility` envelope (`decode_ipc_message_json`) for fail-closed schema versions
## Feature flags
The default dependency graph is wire models plus `RustBackend`. HTTP and ZeroMQ
are opt-in so a consumer of `IpcMessage` / `StimulusBatch` does not compile or
link those stacks.
| *(none)* | yes | Wire models (`IpcMessage`, batches, snapshots) and `RustBackend` |
| `zmq` | no | Binary readout `ZmqIpcBackend`, plus structured `ZmqMessagePublisher` / `ZmqMessageSubscriber` / `ZmqTransportConfig` / `ZmqHybridFlowBackend` (vendored libzmq via `zmq-sys`; needs a C++ compiler). `Send` + `Sync` via mutex-serialized socket ownership; libzmq sockets themselves are `Send` + `!Sync`. |
| `server` | no | `corpus_ipc_server` Axum 0.8 REST binary (`axum` + minimized `tokio`) |
`tower` is not a direct crate dependency. Axum 0.8 depends on it unconditionally,
so it appears only when the optional `server` feature enables Axum. `serde_json`
is a library dependency so the compatibility envelope can inspect `wire_version`
before deserializing a payload. The server binary uses Axum's `Json` extractor
(the `json` feature), which also depends on `serde_json`.
### Three protocol surfaces
These integrations do not share a framing, and none of them is a
guaranteed-delivery bus:
| Structured ZeroMQ | `zmq` | One compact canonical JSON `WireEnvelope` of `IpcMessage` per ZeroMQ message. `ZmqMessagePublisher` binds, `ZmqMessageSubscriber` connects and subscribes to the full stream. `ZmqHybridFlowBackend` maps spikes and embeddings outbound and gradients and eligibility traces inbound through that pair. |
| Binary readout | `zmq` | `ZmqIpcBackend` subscribes to `[i64 tick][N × f32 LE]` packets. It does not implement `HybridFlowBackend` and it does not parse `IpcMessage` JSON. |
| REST | `server` | `corpus_ipc_server` wraps `IpcBackend` (`process_batch`, `initialize`, `save_state`, `reset`). It does not carry `IpcMessage` batches or binary readout frames. |
Structured PUB/SUB is best-effort and at-most-once. There are no
acknowledgements. Frames sent before a subscriber exists (slow joiner), and
frames dropped at the send or receive high-water mark, are gone. `send`
returning success means the local socket accepted the frame, not that a peer
observed it. Control and config messages use that same contract. The default
linger is `0`, so closing the socket discards unsent frames instead of waiting.
A non-blocking receive with nothing queued returns a typed would-block error
(`Ok(None)` on the hybrid-flow receive methods). Malformed JSON, an
incompatible `wire_version`, an invalid payload, multipart or empty framing,
and an unexpected inbound variant return typed errors; unexpected variants are
not discarded quietly. The other supported inbound variant (a gradient while
the caller is receiving traces, or a trace while receiving gradients) is held
for its own method, one message per variant. A second sibling before that
slot is read returns an error and consumes the new frame. A high-water mark of `0` is unlimited, matching libzmq.
Structured sockets also set `ZMQ_MAXMSGSIZE` so one peer frame cannot allocate
without a bound.
`ZmqTransportConfig` carries the endpoint, send and receive high-water marks,
linger, and reconnect interval / maximum. Those socket options are applied
before bind or connect.
The following ZMQ subsections describe the **binary readout** backend.
Installation, the quick start, and protocol ownership come after them.
### ZMQ backend concurrency
`ZmqIpcBackend` is `Send` + `Sync` because `IpcBackend` requires both. libzmq
sockets are **not** safe to share (`zmq(7)` *Thread safety*; `zmq` 0.10's
`Socket` is `Send` + `!Sync`). This crate does not `unsafe impl Sync` on the
socket. The SUB socket is created, subscribed, and connected on the
initializing thread (`initialize` takes `&mut self`) before it is stored in a
`Mutex`. Non-blocking receive and `reset`'s close path take that lock; dropping
the backend closes the socket through exclusive ownership of the struct.
Moving the backend to another thread is supported. `process_batch` still takes
`&mut self`; concurrent process calls need an outer lock, which is how
`corpus_ipc_server` uses `Arc<Mutex<Box<dyn IpcBackend>>>`.
### ZMQ latest-value policy
The ZMQ backend prefers **latest-value** semantics, not queued delivery.
Its SUB socket uses a receive high-water mark (HWM) of 16 packets and does not
enable `ZMQ_CONFLATE`. When the HWM is reached, libzmq silently drops further
publications (it does not evict the oldest queued frames), so a slow consumer
can still observe a stale head-of-queue after a burst. Each `process_batch`
call keeps `DONTWAIT` semantics and drains under a message-count and time
budget (then stops at `EAGAIN`), returning the newest valid frame seen in that
bounded drain (or the cached readout if none arrived). `skipped_readouts()`
counts valid frames deliberately superseded while draining, and
`malformed_readouts()` counts malformed or over-limit frames rejected with
`BackendError` without updating the cache. Neither counter includes packets
libzmq silently drops at its HWM.
### ZMQ reconnect and close behavior
The SUB socket sets `ZMQ_RECONNECT_IVL` / `ZMQ_RECONNECT_IVL_MAX` **before**
`connect`, so recovery after the publisher starts late, stops, or restarts at
the same endpoint is libzmq's own reconnect state machine (the stored
subscribe filter is re-sent on each new connection) — there is no crate-owned
retry thread. Defaults: initial interval **100 ms**, backoff capped at
**5000 ms**; override via `CORPUS_IPC_ZMQ_RECONNECT_IVL_MS` and
`CORPUS_IPC_ZMQ_RECONNECT_IVL_MAX_MS` (positive milliseconds only). `reset()`
plus `initialize()` remains the explicit path for switching endpoints or
forcing manual recovery: `reset` drops the old socket under the backend mutex,
which closes it via `zmq_close` with libzmq's default `ZMQ_LINGER` — a SUB
socket has no outbound queue, so nothing blocks on unsent data. `Drop` of the
backend closes the same way through exclusive ownership.
### ZMQ readout wire format and bounds
Binary SUB packets are **`i64` tick (LE) + `N × f32` readouts (LE)**. An
**88-byte** frame is tick + **20 floats**; length alone cannot distinguish that
from a historical **16 readouts + 4 modulator scores** layout — the backend does
not auto-split or reject all 88-byte packets on that basis (see
`NeuromodulatorSnapshot` rustdoc).
Before resizing the decoded cache, `parse_readout_packet` rejects truncated or
misaligned payloads (`CommunicationError`) and counts above a cap
(`InvalidInput`). Default cap: **1024 floats** (~4 KiB payload), override with
`CORPUS_IPC_ZMQ_MAX_READOUT_FLOATS`. This bounds decoded memory only; libzmq
still allocates the raw received buffer. Non-blocking recv returns the cached
readout on `EAGAIN` (no error).
Golden little-endian fixtures and a SHA-256 manifest live under
`test-vectors/zmq/` (repo-only, not in the crates.io allow-list).
### Known issue: vendored libzmq 4.3.4
This applies to every `zmq` socket this crate opens, not only the binary
readout backend. `zmq-sys` 0.12 builds `zeromq-src` 0.2.6 (libzmq 4.3.4).
That release can SIGSEGV in `ypipe_t<command_t>::write` on newer toolchains
(observed on Ubuntu 26.04 while several sockets connect at once) and
double-frees on pipe shutdown (zeromq/libzmq#4806). This crate shares one
process-wide context but ships no libzmq patch. This repository's CI compiles
libzmq through `scripts/cxx-libzmq-pipefix.sh`, a source patch that is not
part of the published crate. A consumer who hits the crash can copy that
script from the repository and use it as `CXX`, but it rewrites `yqueue.hpp`
in place inside the shared Cargo registry copy of `zeromq-src`, which changes
the build of every project on that machine that uses `zmq-sys` 0.12. Prefer an
isolated `CARGO_HOME` for that build, or restore the registry header
afterward.
## Installation
```toml
[dependencies]
corpus-ipc = "0.2"
# Optional ZMQ backend support
# corpus-ipc = { version = "0.2", features = ["zmq"] }
```
Development / unreleased-main option (tracks `main`, not a registry release):
```toml
[dependencies]
corpus-ipc = { git = "https://github.com/Limen-Neural/corpus-ipc" }
```
Run the REST service from this repo (not pulled in by a library dependency):
```bash
# RustBackend only
CORPUS_IPC_BIND=127.0.0.1:8080 cargo run --release --features server --bin corpus_ipc_server
# With ZMQ backend selectable via CORPUS_IPC_BACKEND_TYPE=zmq
CORPUS_IPC_BIND=127.0.0.1:8080 cargo run --release --features server,zmq --bin corpus_ipc_server
```
The v0.2 REST contract — endpoint request/response bodies, the shared error
envelope, HTTP status classes, and the reset/re-initialize lifecycle — is
specified in [`docs/rest-api.md`](docs/rest-api.md). The router has no
authentication or TLS; a remote bind needs an external access-control
boundary.
## Quick Start
```rust
use corpus_ipc::{BackendType, IpcBackend};
use corpus_ipc::trait_def::BackendFactory;
let mut backend = BackendFactory::create(BackendType::Rust);
backend.initialize(None)?;
let inputs = [0.1_f32, -0.2, 0.3, 0.0];
let outputs = backend.process_batch(&inputs)?;
println!("{}", outputs.len());
# Ok::<(), corpus_ipc::BackendError>(())
```
## Protocol Ownership
`corpus-ipc` is the owner of serialized network schemas for hybrid flow messaging.
Use these re-exports directly from crate root:
```rust
use corpus_ipc::{IpcMessage, SpikeBatch, EmbeddingBatch};
```
### `SpikeBatch` / `TraceBatch` are IPC transport types
These names are **wire payloads** (a session/batch id plus typed
`SpikeEvent` or `TraceData` rows), not in-memory training batches such as
spike-train arrays or unstructured e-prop traces.
The Rust type names stay `SpikeBatch` / `TraceBatch` so serde identifiers
(`IpcMessage::Spikes`, `IpcMessage::EligibilityTraces`) and existing imports
remain compatible. When the collision would be confusing, use the aliases
`IpcSpikeBatch` and `IpcTraceBatch` — they are the same types and the same
wire format.
### Stimulus and neuromodulator ingress
`corpus-ipc` owns the wire schema for stimulus and neuromodulator ingress.
`IpcMessage::Stimuli(StimulusBatch)` and
`IpcMessage::Neuromodulators(NeuromodulatorSnapshot)` carry the payload. A
producer encodes with `encode_canonical_ipc_message` and a consumer decodes
with `decode_ipc_message_json`, which enforces the wire version, instead of bespoke UDP
JSON or a private packet struct. `StimulusBatch` is domain-neutral: its
`values` width is not fixed by this crate (do not encode any one network's
axon/channel count here), and an optional `valid_mask` lets a channel be
marked invalid/missing for a tick instead of silently reading as `0.0`. See
the `StimulusBatch` doc comments for the exact semantics.
### Wire compatibility envelope
Crate semver does not define on-wire acceptance. `WireCompatibility` is the
source of truth for the current and minimum supported **wire** versions
(currently both `1`). Decode hybrid-flow JSON through
`decode_ipc_message_json` so too-old and too-new envelopes return typed
errors before the payload is used. Unversioned tagged `IpcMessage` JSON
(the pre-release git encoding), including JSON-string unit variants such as
`"Ping"`, is still accepted as legacy wire version 1.
Current encoding rules:
- **Unknown fields** on structs and on the envelope object are ignored, so
additive optional fields can be skipped by older readers.
- **Unknown `IpcMessage` variants** fail to deserialize. They never become a
valid default (`IpcMessage` has no `Default` and no serde `other` catch-all).
Bump `WireCompatibility::CURRENT` when the on-wire schema changes in a way
existing decoders cannot ignore. See `CHANGELOG.md` for the full bump rules,
including when to raise `MIN_SUPPORTED`. The compiled example lives on
`decode_ipc_message_json`.
The canonical wire-v1 encoding (sorted keys at every level, finite-float
domain, deterministic bytes) is specified in
[`docs/wire-encoding.md`](docs/wire-encoding.md) and produced by
`encode_canonical_ipc_message`. It is the project wire profile, not RFC 8785 /
JCS.
Golden wire-v1 vectors covering every `IpcMessage` variant — plus
hand-maintained legacy, alias, normalization, and negative fixtures — live
in [`test-vectors/wire-v1/`](test-vectors/wire-v1/README.md) (repository-only,
not packaged). `manifest.json` records each fixture's SHA-256 and decode
expectation for cross-language consumers.
Offline Python and Julia harnesses in [`conformance/`](conformance/README.md)
read those same hashed files and check the Rust decode, f32 normalization,
and negative outcomes. They are not part of the crate build. Linux CI runs
them in a separate job.
## Crate Exports
- Backends and traits:
- `IpcBackend`, `HybridFlowBackend`
- `BackendType` (`Rust`, `ZmqIpc`; deprecated `ZmqRuntime` still selects ZMQ)
- `RustBackend`
- `ZmqIpcBackend` (when `zmq` feature enabled; binary readout; mutex-serialized socket ownership)
- `ZmqMessagePublisher`, `ZmqMessageSubscriber`, `ZmqTransportConfig`, `ZmqHybridFlowBackend` (when `zmq` feature enabled; canonical JSON envelopes)
- `ZmqMessageError`, `FramingFault` (structured-transport errors; `zmq` feature)
- `ProtocolError` (unexpected inbound `IpcMessage` variant)
- Deprecated compatibility aliases: `RuntimeBackend`, `ZmqRuntimeBackend`
- Models:
- `IpcMessage` and all batch/config/trace/gradient payload structs
- `IpcSpikeBatch` / `IpcTraceBatch` aliases for the IPC wire batches
- `StimulusBatch`, `NeuromodulatorSnapshot`
- `ValidationError`, `ValidationKind`, `ProtocolLimits`, `Validate`
- Wire compatibility:
- `WireCompatibility`, `classify_wire_version`, `accept_wire_version`
- `WireEnvelope`, `decode_ipc_message_json`, `encode_ipc_message_json`
- `Compatibility`, `CompatibilityError`, `EnvelopeError`
## Repository
Canonical GitHub home: **[Limen-Neural/corpus-ipc](https://github.com/Limen-Neural/corpus-ipc)**.
`rmems/corpus-ipc` redirects here.
The [GitHub wiki](https://github.com/Limen-Neural/corpus-ipc/wiki) is **enabled**
and is the extra-docs home (architecture, backends, REST server, ecosystem).
In-tree [`docs/`](docs/) is a pointer only — not a second documentation site.
## License
Dual-licensed under either of
- Apache License, Version 2.0 (LICENSE-APACHE-2.0 or <http://www.apache.org/licenses/LICENSE-2.0>)
- MIT License (LICENSE-MIT or <http://opensource.org/licenses/MIT>)
at your option.
## Contribution
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.