rdma-io

Safe Rust bindings for RDMA programming over libibverbs and librdmacm, with async support, tonic gRPC integration, and Quinn QUIC integration.

Features

• Safe RAII wrappers — ProtectionDomain, CompletionQueue, MemoryRegion, QueuePair, CmId, etc. with Arc-based ownership enforcing correct destruction order
• Async stream — AsyncRdmaStream implements tokio::io::AsyncRead + AsyncWrite and futures::AsyncRead + AsyncWrite with dual completion queues for full-duplex I/O
• Transport trait — Generic Transport / TransportBuilder abstraction decoupling RDMA mechanics from consumers; three concrete implementations: SendRecvTransport (Send/Recv verbs), CreditRingTransport (RDMA Write ring + credits), ReadRingTransport (RDMA Write ring + Read flow control)
• Low-level async — AsyncCq (completion queue polling via epoll) and AsyncQp for custom RDMA verb patterns (Send/Recv, RDMA Read/Write, atomics)
• tonic gRPC transport — RdmaConnector and RdmaIncoming for drop-in RDMA transport with tonic, including optional TLS via tonic-tls + OpenSSL
• Quinn QUIC transport — RdmaUdpSocket implements Quinn's AsyncUdpSocket trait, enabling QUIC over RDMA without modifying Quinn
• tonic-h3 gRPC over HTTP/3 — Full stack: tonic gRPC → HTTP/3 → QUIC (Quinn) → RDMA, via tonic-h3 integration
• Generated FFI — rdma-io-sys provides bindings generated with bnd, including wrappers for ibverbs inline functions

Workspace Crates

Quick Start

Async Stream (tokio)

use rdma_io::async_cm::AsyncCmListener;
use rdma_io::async_stream::AsyncRdmaStream;
use rdma_io::send_recv_transport::SendRecvConfig;

let config = SendRecvConfig::stream();

// Server
let listener = AsyncCmListener::bind(&"0.0.0.0:0".parse().unwrap()).unwrap();
let transport = config.accept(&listener).await.unwrap();
let mut server = AsyncRdmaStream::new(transport);

// Client
let addr = "10.0.0.1:9999".parse().unwrap();
let transport = config.connect(&addr).await.unwrap();
let mut client = AsyncRdmaStream::new(transport);
client.write_all(b"hello async rdma").await.unwrap();

tonic gRPC over RDMA

use rdma_io::send_recv_transport::SendRecvConfig;
use rdma_io_tonic::{RdmaConnector, RdmaIncoming};

// Server
let incoming = RdmaIncoming::bind(
    &"0.0.0.0:50051".parse().unwrap(),
    SendRecvConfig::stream(),
)?;
Server::builder()
    .add_service(my_service)
    .serve_with_incoming(incoming).await?;

// Client
let connector = RdmaConnector::new(SendRecvConfig::stream());
let channel = Endpoint::from_static("http://10.0.0.1:50051")
    .connect_with_connector(connector).await?;

Quinn QUIC over RDMA

use rdma_io::send_recv_transport::SendRecvConfig;
use rdma_io_quinn::RdmaUdpSocket;
use quinn::{Endpoint, EndpointConfig};

// Server: bind RDMA socket with transport builder, create Quinn endpoint
let server_socket = Arc::new(
    RdmaUdpSocket::bind(&"0.0.0.0:0".parse().unwrap(), SendRecvConfig::datagram())?
);
let server_endpoint = Endpoint::new_with_abstract_socket(
    EndpointConfig::default(), Some(server_config),
    server_socket, runtime,
)?;
let incoming = server_endpoint.accept().await.unwrap();

// Client: pre-connect RDMA, then create Quinn endpoint
let client_socket = RdmaUdpSocket::bind(&"0.0.0.0:0".parse().unwrap(), SendRecvConfig::datagram())?;
client_socket.connect_to(&server_addr).await?;
let client_endpoint = Endpoint::new_with_abstract_socket(
    EndpointConfig::default(), None,
    Arc::new(client_socket), runtime,
)?;
let connection = client_endpoint.connect(server_addr, "localhost")?.await?;

tonic gRPC over HTTP/3 over RDMA

use rdma_io::send_recv_transport::SendRecvConfig;
use rdma_io_quinn::RdmaUdpSocket;
use tonic_h3::quinn::{H3QuinnAcceptor, H3QuinnConnector};

// Server: RDMA socket → Quinn endpoint → tonic-h3 acceptor
let socket = Arc::new(RdmaUdpSocket::bind(&addr, SendRecvConfig::datagram())?);
let endpoint = Endpoint::new_with_abstract_socket(config, Some(h3_server_config), socket, rt)?;
let acceptor = H3QuinnAcceptor::new(endpoint);
tonic_h3::server::H3Router::new(routes).serve(acceptor).await?;

// Client: pre-connect RDMA → Quinn endpoint → H3 channel → gRPC client
let socket = RdmaUdpSocket::bind(&addr, SendRecvConfig::datagram())?;
socket.connect_to(&server_addr).await?;
let endpoint = Endpoint::new_with_abstract_socket(config, None, Arc::new(socket), rt)?;
let connector = H3QuinnConnector::new(uri.clone(), "localhost".into(), endpoint);
let channel = tonic_h3::H3Channel::new(connector, uri);
let client = GreeterClient::new(channel);

Prerequisites

# Ubuntu/Debian
sudo apt install libibverbs-dev librdmacm-dev rdma-core

# For protobuf compilation (tonic tests)
sudo apt install protobuf-compiler

Software RDMA Providers

For development and testing without RDMA hardware, use one of the software providers:

Both scripts check for kernel modules, load them, create a device, and verify with ibv_devices.

Build

cargo build                    # default (includes tokio async support)
cargo build --no-default-features --features async  # futures-only async, no tokio

To build the rxe kernel module from source (optional):

cmake -B build -DBUILD_RXE=ON
cmake --build build --target rxe

Test

Tests require a software RDMA provider (siw or rxe) and must run single-threaded due to kernel resource contention:

# Set up a provider first
sudo ./scripts/setup-siw.sh

# Run tests
RUST_TEST_THREADS=1 cargo test -p rdma-io-tests -- --nocapture

Documentation

Design documents and background research are in docs/:

CI

CI runs on GitHub Actions with two jobs:

• build-and-test — builds on siw, runs clippy, tests (×5 for flakiness), doc tests
• build-rxe — builds rxe kernel module from source, tests on Soft-RoCE

License

MIT