corpus-ipc

Spike and neuromodulator wire protocol and IPC transports for Limen-Neural SNN runtimes.

corpus-ipc is the schema and transport layer for those runtimes. It provides backend abstractions for local/native execution and optional ZeroMQ IPC, plus canonical wire message models used across services.

Features

• IpcBackend trait for backend-agnostic signal processing (deprecated alias: RuntimeBackend)
• RustBackend reference backend (always available)
• ZmqIpcBackend backend via ZMQ SUB socket (feature zmq; deprecated alias: ZmqRuntimeBackend)
• Optional corpus_ipc_server REST binary (feature server; not compiled for library consumers)
• Canonical protocol models:
  • IpcMessage
  • SpikeBatch, SpikeEvent (IPC wire types; alias IpcSpikeBatch)
  • EmbeddingBatch
  • StimulusBatch for typed, variable-width runtime stimulus ingress
  • GradientBatch, GradientUpdate
  • TraceBatch, TraceData (IPC wire types; alias IpcTraceBatch)
  • ConfigPayload, ConfigValue, BatchMetadata
• HybridFlowBackend trait for message-oriented hybrid transports
• NeuromodulatorSnapshot for typed neuromodulator ingress/readout payloads
• WireCompatibility envelope (decode_ipc_message_json) for fail-closed schema versions

Feature flags

The default dependency graph is wire models plus RustBackend. HTTP and ZeroMQ are opt-in so a consumer of IpcMessage / StimulusBatch does not compile or link those stacks.

tower is not a direct crate dependency. Axum 0.8 depends on it unconditionally, so it appears only when the optional server feature enables Axum. serde_json is a library dependency so the compatibility envelope can inspect wire_version before deserializing a payload. The server binary uses Axum's Json extractor (the json feature), which also depends on serde_json.

Three protocol surfaces

These integrations do not share a framing, and none of them is a guaranteed-delivery bus:

Structured PUB/SUB is best-effort and at-most-once. There are no acknowledgements. Frames sent before a subscriber exists (slow joiner), and frames dropped at the send or receive high-water mark, are gone. send returning success means the local socket accepted the frame, not that a peer observed it. Control and config messages use that same contract. The default linger is 0, so closing the socket discards unsent frames instead of waiting. A non-blocking receive with nothing queued returns a typed would-block error (Ok(None) on the hybrid-flow receive methods). Malformed JSON, an incompatible wire_version, an invalid payload, multipart or empty framing, and an unexpected inbound variant return typed errors; unexpected variants are not discarded quietly. The other supported inbound variant (a gradient while the caller is receiving traces, or a trace while receiving gradients) is held for its own method, one message per variant. A second sibling before that slot is read returns an error and consumes the new frame. A high-water mark of 0 is unlimited, matching libzmq. Structured sockets also set ZMQ_MAXMSGSIZE so one peer frame cannot allocate without a bound.

ZmqTransportConfig carries the endpoint, send and receive high-water marks, linger, and reconnect interval / maximum. Those socket options are applied before bind or connect.

The following ZMQ subsections describe the binary readout backend. Installation, the quick start, and protocol ownership come after them.

ZMQ backend concurrency

ZmqIpcBackend is Send + Sync because IpcBackend requires both. libzmq sockets are not safe to share (zmq(7) Thread safety; zmq 0.10's Socket is Send + !Sync). This crate does not unsafe impl Sync on the socket. The SUB socket is created, subscribed, and connected on the initializing thread (initialize takes &mut self) before it is stored in a Mutex. Non-blocking receive and reset's close path take that lock; dropping the backend closes the socket through exclusive ownership of the struct. Moving the backend to another thread is supported. process_batch still takes &mut self; concurrent process calls need an outer lock, which is how corpus_ipc_server uses Arc<Mutex<Box<dyn IpcBackend>>>.

ZMQ latest-value policy

The ZMQ backend prefers latest-value semantics, not queued delivery. Its SUB socket uses a receive high-water mark (HWM) of 16 packets and does not enable ZMQ_CONFLATE. When the HWM is reached, libzmq silently drops further publications (it does not evict the oldest queued frames), so a slow consumer can still observe a stale head-of-queue after a burst. Each process_batch call keeps DONTWAIT semantics and drains under a message-count and time budget (then stops at EAGAIN), returning the newest valid frame seen in that bounded drain (or the cached readout if none arrived). skipped_readouts() counts valid frames deliberately superseded while draining, and malformed_readouts() counts malformed or over-limit frames rejected with BackendError without updating the cache. Neither counter includes packets libzmq silently drops at its HWM.

ZMQ reconnect and close behavior

The SUB socket sets ZMQ_RECONNECT_IVL / ZMQ_RECONNECT_IVL_MAX before connect, so recovery after the publisher starts late, stops, or restarts at the same endpoint is libzmq's own reconnect state machine (the stored subscribe filter is re-sent on each new connection) — there is no crate-owned retry thread. Defaults: initial interval 100 ms, backoff capped at 5000 ms; override via CORPUS_IPC_ZMQ_RECONNECT_IVL_MS and CORPUS_IPC_ZMQ_RECONNECT_IVL_MAX_MS (positive milliseconds only). reset() plus initialize() remains the explicit path for switching endpoints or forcing manual recovery: reset drops the old socket under the backend mutex, which closes it via zmq_close with libzmq's default ZMQ_LINGER — a SUB socket has no outbound queue, so nothing blocks on unsent data. Drop of the backend closes the same way through exclusive ownership.

ZMQ readout wire format and bounds

Binary SUB packets are i64 tick (LE) + N × f32 readouts (LE). An 88-byte frame is tick + 20 floats; length alone cannot distinguish that from a historical 16 readouts + 4 modulator scores layout — the backend does not auto-split or reject all 88-byte packets on that basis (see NeuromodulatorSnapshot rustdoc).

Before resizing the decoded cache, parse_readout_packet rejects truncated or misaligned payloads (CommunicationError) and counts above a cap (InvalidInput). Default cap: 1024 floats (~4 KiB payload), override with CORPUS_IPC_ZMQ_MAX_READOUT_FLOATS. This bounds decoded memory only; libzmq still allocates the raw received buffer. Non-blocking recv returns the cached readout on EAGAIN (no error).

Golden little-endian fixtures and a SHA-256 manifest live under test-vectors/zmq/ (repo-only, not in the crates.io allow-list).

Known issue: vendored libzmq 4.3.4

This applies to every zmq socket this crate opens, not only the binary readout backend. zmq-sys 0.12 builds zeromq-src 0.2.6 (libzmq 4.3.4). That release can SIGSEGV in ypipe_t<command_t>::write on newer toolchains (observed on Ubuntu 26.04 while several sockets connect at once) and double-frees on pipe shutdown (zeromq/libzmq#4806). This crate shares one process-wide context but ships no libzmq patch. This repository's CI compiles libzmq through scripts/cxx-libzmq-pipefix.sh, a source patch that is not part of the published crate. A consumer who hits the crash can copy that script from the repository and use it as CXX, but it rewrites yqueue.hpp in place inside the shared Cargo registry copy of zeromq-src, which changes the build of every project on that machine that uses zmq-sys 0.12. Prefer an isolated CARGO_HOME for that build, or restore the registry header afterward.

Installation

[dependencies]
corpus-ipc = "0.2"

# Optional ZMQ backend support
# corpus-ipc = { version = "0.2", features = ["zmq"] }

Development / unreleased-main option (tracks main, not a registry release):

[dependencies]
corpus-ipc = { git = "https://github.com/Limen-Neural/corpus-ipc" }

Run the REST service from this repo (not pulled in by a library dependency):

# RustBackend only
CORPUS_IPC_BIND=127.0.0.1:8080 cargo run --release --features server --bin corpus_ipc_server

# With ZMQ backend selectable via CORPUS_IPC_BACKEND_TYPE=zmq
CORPUS_IPC_BIND=127.0.0.1:8080 cargo run --release --features server,zmq --bin corpus_ipc_server

The v0.2 REST contract — endpoint request/response bodies, the shared error envelope, HTTP status classes, and the reset/re-initialize lifecycle — is specified in docs/rest-api.md. The router has no authentication or TLS; a remote bind needs an external access-control boundary.

Quick Start

use corpus_ipc::{BackendType, IpcBackend};
use corpus_ipc::trait_def::BackendFactory;

let mut backend = BackendFactory::create(BackendType::Rust);
backend.initialize(None)?;

let inputs = [0.1_f32, -0.2, 0.3, 0.0];
let outputs = backend.process_batch(&inputs)?;

println!("{}", outputs.len());
# Ok::<(), corpus_ipc::BackendError>(())

Protocol Ownership

corpus-ipc is the owner of serialized network schemas for hybrid flow messaging.

Use these re-exports directly from crate root:

use corpus_ipc::{IpcMessage, SpikeBatch, EmbeddingBatch};

SpikeBatch / TraceBatch are IPC transport types

These names are wire payloads (a session/batch id plus typed SpikeEvent or TraceData rows), not in-memory training batches such as spike-train arrays or unstructured e-prop traces.

The Rust type names stay SpikeBatch / TraceBatch so serde identifiers (IpcMessage::Spikes, IpcMessage::EligibilityTraces) and existing imports remain compatible. When the collision would be confusing, use the aliases IpcSpikeBatch and IpcTraceBatch — they are the same types and the same wire format.

Stimulus and neuromodulator ingress

corpus-ipc owns the wire schema for stimulus and neuromodulator ingress. IpcMessage::Stimuli(StimulusBatch) and IpcMessage::Neuromodulators(NeuromodulatorSnapshot) carry the payload. A producer encodes with encode_canonical_ipc_message and a consumer decodes with decode_ipc_message_json, which enforces the wire version, instead of bespoke UDP JSON or a private packet struct. StimulusBatch is domain-neutral: its values width is not fixed by this crate (do not encode any one network's axon/channel count here), and an optional valid_mask lets a channel be marked invalid/missing for a tick instead of silently reading as 0.0. See the StimulusBatch doc comments for the exact semantics.

Wire compatibility envelope

Crate semver does not define on-wire acceptance. WireCompatibility is the source of truth for the current and minimum supported wire versions (currently both 1). Decode hybrid-flow JSON through decode_ipc_message_json so too-old and too-new envelopes return typed errors before the payload is used. Unversioned tagged IpcMessage JSON (the pre-release git encoding), including JSON-string unit variants such as "Ping", is still accepted as legacy wire version 1.

Current encoding rules:

• Unknown fields on structs and on the envelope object are ignored, so additive optional fields can be skipped by older readers.
• Unknown IpcMessage variants fail to deserialize. They never become a valid default (IpcMessage has no Default and no serde other catch-all).

Bump WireCompatibility::CURRENT when the on-wire schema changes in a way existing decoders cannot ignore. See CHANGELOG.md for the full bump rules, including when to raise MIN_SUPPORTED. The compiled example lives on decode_ipc_message_json.

The canonical wire-v1 encoding (sorted keys at every level, finite-float domain, deterministic bytes) is specified in docs/wire-encoding.md and produced by encode_canonical_ipc_message. It is the project wire profile, not RFC 8785 / JCS.

Golden wire-v1 vectors covering every IpcMessage variant — plus hand-maintained legacy, alias, normalization, and negative fixtures — live in test-vectors/wire-v1/ (repository-only, not packaged). manifest.json records each fixture's SHA-256 and decode expectation for cross-language consumers.

Offline Python and Julia harnesses in conformance/ read those same hashed files and check the Rust decode, f32 normalization, and negative outcomes. They are not part of the crate build. Linux CI runs them in a separate job.

Crate Exports

• Backends and traits:
  • IpcBackend, HybridFlowBackend
  • BackendType (Rust, ZmqIpc; deprecated ZmqRuntime still selects ZMQ)
  • RustBackend
  • ZmqIpcBackend (when zmq feature enabled; binary readout; mutex-serialized socket ownership)
  • ZmqMessagePublisher, ZmqMessageSubscriber, ZmqTransportConfig, ZmqHybridFlowBackend (when zmq feature enabled; canonical JSON envelopes)
  • ZmqMessageError, FramingFault (structured-transport errors; zmq feature)
  • ProtocolError (unexpected inbound IpcMessage variant)
  • Deprecated compatibility aliases: RuntimeBackend, ZmqRuntimeBackend
• Models:
  • IpcMessage and all batch/config/trace/gradient payload structs
  • IpcSpikeBatch / IpcTraceBatch aliases for the IPC wire batches
  • StimulusBatch, NeuromodulatorSnapshot
  • ValidationError, ValidationKind, ProtocolLimits, Validate
• Wire compatibility:
  • WireCompatibility, classify_wire_version, accept_wire_version
  • WireEnvelope, decode_ipc_message_json, encode_ipc_message_json
  • Compatibility, CompatibilityError, EnvelopeError

Repository

Canonical GitHub home: Limen-Neural/corpus-ipc. rmems/corpus-ipc redirects here.

The GitHub wiki is enabled and is the extra-docs home (architecture, backends, REST server, ecosystem). In-tree docs/ is a pointer only — not a second documentation site.

License

Dual-licensed under either of

• Apache License, Version 2.0 (LICENSE-APACHE-2.0 or http://www.apache.org/licenses/LICENSE-2.0)
• MIT License (LICENSE-MIT or http://opensource.org/licenses/MIT)

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.