Crate ibverbs

Rust API wrapping the ibverbs RDMA library.

libibverbs is a library that allows userspace processes to use RDMA “verbs” to perform high-throughput, low-latency network operations for both Infiniband (according to the Infiniband specifications) and iWarp (iWARP verbs specifications). It handles the control path of creating, modifying, querying and destroying resources such as Protection Domains, Completion Queues, Queue-Pairs, Shared Receive Queues, Address Handles, and Memory Regions. It also handles sending and receiving data posted to QPs and SRQs, and getting completions from CQs using polling and completions events.

A good place to start is to look at the programs in examples/, and the upstream C examples. You can test RDMA programs on modern Linux kernels even without specialized RDMA hardware by using SoftRoCE.

For the detail-oriented

The control path is implemented through system calls to the uverbs kernel module, which further calls the low-level HW driver. The data path is implemented through calls made to low-level HW library which, in most cases, interacts directly with the HW provides kernel and network stack bypass (saving context/mode switches) along with zero copy and an asynchronous I/O model.

iWARP ethernet NICs support RDMA over hardware-offloaded TCP/IP, while InfiniBand is a general high-throughput, low-latency networking technology. InfiniBand host channel adapters (HCAs) and iWARP NICs commonly support direct hardware access from userspace (kernel bypass), and libibverbs supports this when available.

For more information on RDMA verbs, see the InfiniBand Architecture Specification vol. 1, especially chapter 11, and the RDMA Consortium’s RDMA Protocol Verbs Specification. See also the upstream libibverbs/verbs.h file for the original C definitions, as well as the manpages for the ibv_* methods.

Library dependency

libibverbs is usually available as a free-standing library package. It used to be self-contained, but has recently been adopted into rdma-core. cargo will automatically build the necessary library files and place them in vendor/rdma-core/build/lib. If a system-wide installation is not available, those library files can be used instead by copying them to /usr/lib, or by adding that path to the dynamic linking search path.

Thread safety

All interfaces are Sync and Send since the underlying ibverbs API is thread safe.

Documentation

Much of the documentation of this crate borrows heavily from the excellent posts over at RDMAmojo. If you are going to be working a lot with ibverbs, chances are you will want to head over there. In particular, this overview post may be a good place to start.

Modules

ibv_qp_type

ibv_wc_opcode

ibv_wc_status

Structs

CompletionQueue

A completion queue that allows subscribing to the completion of queued sends and receives.

Context

An RDMA context bound to a device.

Device

An RDMA device.

DeviceList

List of available RDMA devices.

DeviceListIter

Iterator over a DeviceList.

Gid

A Global identifier for ibv.

GidEntry

A Global identifier entry for ibv.

Guid

A Global unique identifier for ibv.

MemoryRegion

A memory region that has been registered for use with RDMA.

PreparedQueuePair

An allocated but uninitialized QueuePair.

ProtectionDomain

A protection domain for a device’s context.

QueuePair

A fully initialized and ready QueuePair.

QueuePairBuilder

An unconfigured QueuePair.

QueuePairEndpoint

An identifier for the network endpoint of a QueuePair.

RemoteKey

A key that authorizes direct memory access to a memory region.

ibv_access_flags

Access flags for use with QueuePair and MemoryRegion.

ibv_wc

An ibverb work completion.

Functions

devices

Get list of available RDMA devices.

Type Aliases

ibv_gid_type

Direct access to low-level libverbs FFI.