rdmatop

htop, but for RDMA traffic — a real-time TUI monitor for RDMA network interfaces.

Monitors per-device throughput (Gbps, packets/s, drops), RDMA read/write counters, retransmits, health events, and shows which processes are using each RDMA device — all via RDMA netlink, the same interface used by rdma statistic.

Blogs

• rdmatop: Cross-Provider htop for RDMA Traffic (2026-06-15)
• NVSHMEM Multi-NIC Support with AWS EFA (2026-03-27)

Requirements

• Linux (netlink-based — macOS/Windows are not supported)
• RDMA-capable NICs (e.g., Mellanox/NVIDIA ConnectX, AWS EFA)

Installation

Ubuntu (PPA)

On Ubuntu 22.04 (jammy), 24.04 (noble), or 26.04 (resolute) — amd64 and arm64:

sudo add-apt-repository ppa:crazyguitar/rdmatop
sudo apt update
sudo apt install rdmatop

Cargo

cargo install rdmatop

From source

make         # cargo build
make install # cargo install

Usage

rdmatop

Perfetto recording

Press r in the TUI to start recording and r again to stop. rdmatop captures every device's tx/rx Gbps and packets/s per interval and writes a Chrome-JSON trace (rdmatop-<unix_timestamp>.json, in the current directory) you can drag into ui.perfetto.dev — each device/port becomes its own set of counter tracks. Timestamps are relative to when you pressed r, so the trace spans exactly your record window.

PyTorch profiler

rdmatop can run inside the training process as a Kineto child profiler, so RDMA counter tracks land in the same trace torch.profiler writes:

import torch
from torch.profiler import ProfilerActivity, profile

import rdmatop.kineto

rdmatop.kineto.enable()
with profile(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA]) as prof:
    train_step()
prof.export_chrome_trace("trace.json")

The shim links against the installed torch (>= 2.9), so it is built from a checkout with cargo and a C++ compiler on PATH. PyTorch selects the required C++ standard (C++17 or C++20, depending on its version). Rebuild the shim after changing PyTorch versions:

pip install "setuptools>=64" "torch>=2.9"
pip install --no-build-isolation -e ./python

On older Kineto versions without native counters, enable() wraps torch.profiler.profile.export_chrome_trace() to convert rdmatop's marked events into counter tracks. This also supports gzip exports and tensorboard_trace_handler; raw Kineto exports retain zero-duration events. Newer versions emit native counters and need no export wrapper.

VizTracer

pip install "viztracer>=1.1.1"
viztracer --plugins rdmatop.viztracer -- my_script.py

from viztracer import VizTracer

with VizTracer(plugins=["rdmatop.viztracer"], output_file="trace.json"):
    train_step()

viztracer --plugins "rdmatop.viztracer --interval 1 --devices mlx5_0,mlx5_1" -- my_script.py

Examples

Use rdmatop to monitor RDMA traffic while running GPU communication benchmarks:

• PyTorch — intranode NVLink/XGMI traffic
• IB Perftest — two-node ib_write_bw benchmark
• UCX Perftest — two-node ucx_perftest bandwidth / latency
• NCCL — collective communication
• NIXL — point-to-point KV cache transfer
• NVSHMEM — one-sided GPU communication
• PPLX Kernels — MoE all-to-all dispatch/combine
• UCCL — DeepEP-compatible expert-parallel dispatch/combine
• RDMA Statistics — shell-based RDMA stats
• Kubernetes — DaemonSet deployment for Kubernetes

How It Works

1. Device enumeration — RDMA_NLDEV_CMD_GET via netlink to discover all RDMA devices
2. HW counters — RDMA_NLDEV_CMD_STAT_GET per device/port, same as rdma statistic show
3. Process detection — RDMA_NLDEV_CMD_RES_QP_GET to map QPs → PIDs, enriched with /proc data
4. Throughput — Two snapshots per interval, delta / elapsed for rates

Contributing

See CONTRIBUTING.md for build/test instructions, design ground rules, and how to submit changes.

License

Apache-2.0