# Lecture Note 07 (L15, L16): DMA, IOMMU & GPU Memory; NUMA & HPC Optimization

**Combines:** Lecture L15 (DMA, IOMMU & GPU Memory Management) and Lecture L16 (NUMA Topology & HPC Memory Optimization).

---

## How This Note Is Organized

1. **Part 1 — DMA & IOMMU:** Direct Memory Access; cache coherency (coherent vs streaming); DMA-BUF; IOMMU and security; VFIO; GPU memory and zero-copy.
2. **Part 2 — NUMA:** Non-uniform memory access; first-touch and placement; memory policies (bind, preferred, interleave); numactl and libnuma; AutoNUMA; multi-GPU and CPU–GPU affinity.

---

# Part 1: DMA, IOMMU & GPU Memory Management

**Context:** Devices move data to/from RAM without CPU copies. The CPU must not see stale data (cache coherency). The IOMMU translates device addresses and restricts access; DMA-BUF shares one buffer across CPU, GPU, camera, display.

---

## DMA (Direct Memory Access)

- Device (NIC, NVMe, GPU, camera ISP) transfers data to/from system RAM autonomously. CPU programs descriptor (src, dst, length); device runs transfer; completion via IRQ or poll.
- **Without DMA:** Device → CPU copy → RAM. **With DMA:** Device → DMA engine → RAM; CPU is free after submitting descriptor.

---

## Cache Coherency: Coherent vs Streaming

- **Coherent DMA:** `dma_alloc_coherent()` — uncached or hardware-coherent; CPU and device always see same data; no explicit sync. Use for small control/descriptor regions.
- **Streaming DMA:** CPU uses cached memory; driver **synchronizes** explicitly:
  - **DMA_TO_DEVICE:** CPU wrote data → `dma_map_single()` flushes cache → device reads.
  - **DMA_FROM_DEVICE:** Device will write → after transfer, `dma_unmap_single()` invalidates cache → CPU reads. Between map and unmap the buffer is "owned" by the device — CPU must not touch it.
- **dma_map_sg** / **dma_unmap_sg** for scatter-gather (fragmented buffers). Direction controls which cache op is used; wrong direction = silent corruption.

---

## IOMMU (Input-Output MMU)

- Sits between devices and memory bus. Translates **IOVA** (I/O Virtual Address) to physical address using per-device/group page tables.
- **Without IOMMU:** Device can DMA to any physical address (security risk). **With IOMMU:** Only mapped IOVAs are allowed; unmapped access → fault.
- **Implementations:** Intel VT-d, AMD-Vi, ARM SMMU (Jetson Orin). **IOMMU groups:** Devices behind same translation unit; for VFIO passthrough, whole group is assigned together.
- Drivers usually use **dma_map_*** which uses IOMMU automatically; low-level `iommu_map`/`iommu_unmap` in framework code.

---

## DMA-BUF & Zero-Copy Pipeline

- **DMA-BUF:** Kernel abstraction to share one DMA buffer across subsystems (CPU, GPU, camera ISP, display). **Exporter** allocates and exports; **importer** attaches and maps for its device.
- **Lifecycle:** Exporter creates buffer, gets fd; fd passed (e.g. Unix socket); importer `dma_buf_get(fd)`, `dma_buf_attach`, `dma_buf_map_attachment` → sg_table with IOVAs for that device.
- **Zero-copy:** V4L2 camera → DMA-BUF fd → CUDA importer → same physical pages for inference → display. No CPU copy; sync via DMA fence (`dma_fence_wait`). V4L2 supports `V4L2_MEMORY_DMABUF`; userspace passes fd in buffer.

---

## GPU Memory & Unified Memory

- Discrete GPU: dedicated VRAM; driver manages allocations and CPU↔GPU copies. **Unified memory (e.g. Jetson):** CPU and GPU share physical RAM; one address space; no explicit copy for shared buffers, but bandwidth and latency depend on placement.
- **Resizable BAR (SAM):** GPU BAR1 can cover full VRAM; CPU can address all GPU memory (important for GPUDirect Storage, zero-copy).

---

# Part 2: NUMA Topology & HPC Memory Optimization

**Context:** On multi-socket machines, memory attached to socket 0 is "local" to CPUs on socket 0 and "remote" to socket 1. Remote access has higher latency and lower bandwidth; placement of data and threads matters.

---

## NUMA Architecture

- Each socket (NUMA node) has local DRAM (lower latency, full bandwidth). Access to another node's DRAM goes over interconnect (QPI/UPI, Infinity Fabric) — ~2× latency, ~half bandwidth.
- **Discovery:** `numactl --hardware`, `lstopo`, `numastat`, `/sys/devices/system/node/nodeN/distance`. Distance matrix: local = 10; remote often 20–40.

---

## First-Touch and Placement

- **Default (first-touch):** Page is allocated on the node of the CPU that first **faults** it. If main thread on node 0 initializes a buffer later used only by workers on node 1, every access is remote — silent slowdown.
- **Fix:** Allocate (and fault) on the same node that will use the data: pin thread to node, set mempolicy, then malloc + memset; or use `numa_alloc_onnode()` / `mbind(..., MPOL_MF_MOVE)` to move pages.

---

## Memory Policies & numactl

- **MPOL_DEFAULT:** First-touch. **MPOL_BIND:** Only listed nodes; fail if full. **MPOL_PREFERRED:** Prefer one node; fallback. **MPOL_INTERLEAVE:** Round-robin across nodes (bandwidth-bound).
- **set_mempolicy** (process); **mbind** (VMA; MPOL_MF_MOVE migrates existing pages).
- **numactl:** `--cpunodebind=0 --membind=0 ./app` (bind CPU and memory to node 0); `--interleave=all` (interleave); `--preferred=1` (prefer node 1).

---

## libnuma & AutoNUMA

- **libnuma:** `numa_node_of_cpu(sched_getcpu())`, `numa_alloc_onnode()`, `numa_bind()`, `numa_set_membind()`, `numa_set_interleave_mask()`.
- **AutoNUMA:** Kernel migrates "hot" pages to the node of the accessing CPU. Overhead (scan, TLB shootdown, migration); can cause tail latency spikes. **Recommendation:** Disable (`numa_balancing=0`) on RT and latency-sensitive inference; use explicit numactl/mbind.

---

## Multi-GPU & CPU–GPU Affinity

- GPUs are attached to one socket's PCIe root. CPU↔GPU traffic from the other socket crosses interconnect. Use `nvidia-smi topo -m` to see topology; bind process and memory to the node that owns the GPU(s) used.

---

## Summary Tables

**DMA:** Coherent = uncached/coherent, no sync. Streaming = map (flush/invalidate) → device uses → unmap (invalidate/flush); CPU must not touch between map/unmap.

**IOMMU:** IOVA→PA; per-device/group; restricts DMA; VFIO exposes to userspace for passthrough.

**NUMA:** Local vs remote latency/bandwidth; first-touch; bind/preferred/interleave; numactl; disable AutoNUMA for RT.

---

## AI Hardware Connection

- **DMA-BUF + V4L2:** Zero-copy camera → GPU; fd over IPC. **Streaming DMA** direction must be correct (TO_DEVICE / FROM_DEVICE) to avoid corruption.
- **IOMMU:** Security and isolation for GPU/VFIO; required for safe passthrough.
- **NUMA:** Pin inference and weights to same node as GPU; first-touch on wrong node halves effective bandwidth. **numastat -p** to verify; disable AutoNUMA on inference servers.
- **Multi-GPU:** Place processes and allocations on the node that owns the target GPU; use topology tools to avoid cross-socket PCIe.

---

*Combines Lectures L15, L16 (DMA, IOMMU, GPU Memory; NUMA, HPC Optimization).*
