Crate netidx

Crate netidx 

Source
Expand description

§Netidx - Real-time data sharing for distributed systems

High-performance pub/sub middleware without the message broker.

Netidx lets you publish and subscribe to live data across your network using a simple, hierarchical namespace. Data flows directly between publishers and subscribers over TCP—no central broker bottleneck.

§Why Netidx?

  • Fast: Direct connections, zero-copy where possible, handles tens of millions of updates/sec
  • Discoverable: Hierarchical namespace like a filesystem, browse and query what exists
  • Flexible: Optional persistence (netidx-container), event logging (netidx-archive)
  • Secure: Built-in Kerberos v5 and TLS with centralized authorization
  • Production-tested: Years in production in financial trading systems

§Quick Start

Publishing data:

use netidx::{
    publisher::{PublisherBuilder, Value, BindCfg, DesiredAuth},
    config::Config, path::Path,
};
use tokio::time;
use std::time::Duration;
let cfg = Config::load_default()?;
let publisher = PublisherBuilder::new(cfg)
    .desired_auth(DesiredAuth::Anonymous)
    .bind_cfg(Some("192.168.0.0/16".parse()?))
    .build()
    .await?;

let temp = publisher.publish(
    Path::from("/hw/cpu-temp"),
    get_cpu_temp()
)?;

loop {
    time::sleep(Duration::from_millis(500)).await;
    let mut batch = publisher.start_batch();
    temp.update(&mut batch, get_cpu_temp());
    batch.commit(None).await;
}

Subscribing to data:

use netidx::{
    subscriber::{Subscriber, UpdatesFlags, DesiredAuth},
    config::Config, path::Path,
};
use futures::{prelude::*, channel::mpsc};
let cfg = Config::load_default()?;
let subscriber = Subscriber::new(cfg, DesiredAuth::Anonymous)?;
let temp = subscriber.subscribe(Path::from("/hw/cpu-temp"));
temp.wait_subscribed().await;

println!("Current: {:?}", temp.last());

let (tx, mut rx) = mpsc::channel(10);
temp.updates(UpdatesFlags::empty(), tx);
while let Some(mut batch) = rx.next().await {
    for (_, value) in batch.drain(..) {
        println!("Updated: {:?}", value);
    }
}

§How It Works

Netidx has three components:

  • Resolver Server: Directory service mapping paths to publisher addresses
  • Publishers: Create values and serve subscribers directly
  • Subscribers: Connect directly to publishers for live data

Unlike traditional message brokers, the resolver only stores addresses, not data. This eliminates the broker as a bottleneck and single point of failure.

§Key Features

  • Direct TCP connections - No broker bottleneck, connection pooling
  • Type-safe values - Rich types including primitives, strings, bytes, arrays, maps
  • User-defined types - Publish your own custom types by implementing a few traits
  • Bi-directional - Subscribers can write values back to publishers
  • Durable subscriptions - Automatic reconnection and state recovery
  • Authorization - Centralized permissions with Kerberos v5 or TLS
  • Discovery - Browse namespace, glob patterns, structural queries

§Optional Components

§Documentation

§Architecture

Values are transmitted as a reliable, ordered stream over TCP (like the connection itself, but with publisher::Value as the unit instead of bytes). Published values always have a current value that new subscribers receive immediately, followed by live updates.

For best discoverability, structure your data hierarchically with multiple published values rather than complex nested structures. This is both efficient and makes your system browsable.

Re-exports§

pub use netidx_netproto as protocol;
pub use poolshark as pool;

Modules§

config
Configuration file loading and management.
pack
Efficient binary serialization with forward compatibility.
path
Hierarchical path handling and manipulation.
publisher
Publish values to subscribers.
resolver_client
Client for querying the resolver server.
resolver_server
Resolver server for mapping paths to publishers.
subscriber
Subscribe to published values.
tls
TLS configuration and utilities.
utils
Utility types and functions.

Structs§

InternalOnly
A self-contained netidx setup for testing or standalone use.