# rdmaprobe - hardware-validation findings (2026-05-22)

Validated on two real ConnectX HCAs (GH200 arm64 RoCE, 8x V100 amd64 RoCE).

## What's validated
- The eBPF program **verifier-loads** on a real amd64 kernel (ringbuf,
  bounded loop, `bpf_probe_read_user` all accepted).
- `Probe.Attach` **successfully attaches** uprobe+uretprobe to an exported
  libibverbs symbol on real hardware.
- `internal/rdma` D3 sysfs counter reader validated on **both** arm64 and
  amd64 (RoCE link layer, all five counters parse).

## CRITICAL: the poll_cq target in this package is INFEASIBLE on stock distros
`bpf/rdma_verbs.bpf.c` and `verbs.go` currently target the provider
`poll_cq` to read per-completion `ibv_wc.status`. On a stock distro this
cannot be attached:
- `libmlx5.so` is **stripped** - no static `mlx5_poll_cq` symbol (by name or
  to derive an offset).
- `ibv_poll_cq` is `static inline` in `<infiniband/verbs.h>` - inlined into
  the caller, not an exported symbol anywhere.

## The feasible verbs target: `ibv_get_async_event`
Exported (`T`) in `libibverbs.so`. Carries the fabric/QP events D1/D2 want
(`enum ibv_event_type`: `IBV_EVENT_PORT_ERR`, `IBV_EVENT_PORT_ACTIVE`,
`IBV_EVENT_QP_FATAL`, `IBV_EVENT_DEVICE_FATAL`, ...) via
`struct ibv_async_event { union {...} element; enum ibv_event_type event_type; }`
(event_type at offset 8; struct size 16). Uprobe+uretprobe attach was
validated against it on real hardware. `ibv_modify_qp` / `ibv_create_qp`
are also exported for QP-lifecycle if needed.

## Pivot DONE (2026-05-22)
The eBPF + Go were rewritten from poll_cq/ibv_wc to ibv_get_async_event:
- `bpf/rdma_verbs.bpf.c`: uprobe/uretprobe on ibv_get_async_event; entry
  stashes `struct ibv_async_event*`, return reads `event_type` (@8) +
  `element` (@0) and emits an rdma_async_event. Compiles arm64+x86.
- `eventtype.go`: ibv_event_type decoder (Name/IsError/IsFabricError; the
  last scoped to PORT_ERR/PATH_MIG_ERR/DEVICE_FATAL). Replaced wcstatus.go.
- `verbs.go`: Event{TimestampNs,Element,PID,TID,EventType}; DefaultSymbol =
  "ibv_get_async_event"; attaches via libibverbs path from discovery.
- `discovery.go`: finds libibverbs in /proc/<pid>/maps.
All built both arches + unit-tested.

## Live capture VALIDATED (2026-05-22, 8x V100 / ConnectX-5 RoCE, amd64)
A self-healing eno1 flap (down->up->netplan apply) generated real port
events while a libibverbs consumer + this probe ran. The probe captured and
correctly decoded **IBV_EVENT_PORT_ACTIVE** (type 9) and 24x
**IBV_EVENT_GID_CHANGE** (type 18) end-to-end (uprobe fires -> reads
event_type -> ringbuf -> userspace decode + fabric classification). The
capture path is proven on real hardware.
- **Follow-up nit:** the consumer also logged one IBV_EVENT_PORT_ERR (type
  10, link-down) that did NOT appear in the probe log - a single
  transient-window miss at the instant of link-down. Worth a look (ringbuf
  reserve race?), not a blocker; PORT_ACTIVE (also a port event) and all GID
  changes were captured.

## Wired into the trace command (commit 90c5e62)
`--rdma-verbs` attaches the probe to a libibverbs workload, reads async
events into a cumulative buffer, emits ingero.rdma.async_event_total
(rdma.event_type + rdma.fabric_error). D3 sysfs counters emit
ingero.rdma.<counter>. Both degrade to no-ops without an HCA / CAP_BPF.

## Shelved
Per-WC-completion error capture (RETRY_EXC_ERR etc.): infeasible by uprobe on
stripped stock libmlx5; the sysfs counters (D3) + async events cover the
measured fabric signal.
