# Lecture 2: Processes, task_struct & the Linux Process Model

## Overview

A running AI system is not a single program — it is a collection of competing processes that must share a CPU, memory, and hardware devices without interfering with each other. The core challenge this lecture addresses is: how does the Linux kernel track every running program and safely multiplex the hardware among them? The mental model to carry forward is that every running entity in Linux — whether a process, a thread, or a kernel worker — is represented by one structure: `task_struct`. Understanding this structure is understanding how the kernel sees your code. For an AI hardware engineer, this matters because scheduler class, CPU affinity, cgroup membership, and memory layout are all fields in `task_struct`, and tuning inference pipeline performance means knowing which knobs map to which fields.

---

## The Process Abstraction

A **process** is a program in execution. It combines three orthogonal components:

- **Virtual CPU**: register state (PC, SP, general-purpose registers) saved in `task_struct` during preemption
- **Virtual memory**: address space — text, data, heap, stack, and memory-mapped regions, described by `mm_struct`
- **Resources**: file descriptor table, signal handlers, sockets, cgroup membership — all reachable from `task_struct`

Threads are processes that share `mm_struct` and `files_struct` but have independent stacks and register state. Linux makes no kernel distinction between "process" and "thread" — both are represented by `task_struct`.

> **Key Insight:** Linux has no separate "thread" concept at the kernel level. A thread is simply a task that shares its `mm_struct` (address space) with another task. This design simplifies the scheduler but means every thread has its own `task_struct`, its own PID (visible via `gettid()`), and its own scheduler entity. When you pin CPU affinity for a thread, you are writing to that thread's `task_struct.cpus_mask`.

---

## task_struct Key Fields

`task_struct` is defined in `include/linux/sched.h`. It is large (~5 KB); only the fields relevant to AI/embedded work are listed here.

```
task_struct — The kernel's representation of a running task
┌─────────────────────────────────────────────────────────┐
│  pid       — unique thread ID (gettid())                │
│  tgid      — thread group ID; all threads share this    │
│              (getpid() returns tgid)                    │
│  state     — TASK_RUNNING / TASK_INTERRUPTIBLE / etc.   │
├─────────────────────────────────────────────────────────┤
│  mm ──────────────────────────────> mm_struct           │
│                                     (virtual address    │
│                                      space, page table) │
├─────────────────────────────────────────────────────────┤
│  files ────────────────────────────> files_struct       │
│                                     (open FD table;     │
│                                      shared by threads) │
├─────────────────────────────────────────────────────────┤
│  sched_class ──> rt / fair / dl / idle / stop           │
│  se     — CFS/EEVDF entity (vruntime, load weight)      │
│  rt     — RT entity (static priority, time slice)       │
│  dl     — DEADLINE entity (runtime, deadline, period)   │
├─────────────────────────────────────────────────────────┤
│  cgroups ──────────────────────────> css_set            │
│  cpus_mask  — CPU affinity bitmask                      │
└─────────────────────────────────────────────────────────┘
```

| Field | Type | Purpose |
|---|---|---|
| `pid` | `pid_t` | Process ID — unique per thread in the system |
| `tgid` | `pid_t` | Thread group ID — shared across all threads; returned by `getpid()` |
| `state` | `unsigned int` | Current run state (TASK_RUNNING, TASK_INTERRUPTIBLE, etc.) |
| `mm` | `struct mm_struct *` | Virtual memory descriptor; NULL for kernel threads |
| `fs` | `struct fs_struct *` | Filesystem root and working directory |
| `files` | `struct files_struct *` | Open file descriptor table |
| `signal` | `struct signal_struct *` | Signal handlers, pending signals, process group |
| `sched_class` | pointer | Which scheduler class handles this task |
| `se` | `struct sched_entity` | CFS/EEVDF entity: vruntime, load weight, slice |
| `rt` | `struct sched_rt_entity` | RT entity: static priority, time slice |
| `dl` | `struct sched_dl_entity` | DEADLINE entity: runtime, deadline, period budgets |
| `cgroups` | `struct css_set *` | cgroup v2 membership; pointer to CSS set |
| `cpus_mask` | `cpumask_t` | Allowed CPU affinity; set via `sched_setaffinity()` |

---

## Process States

Understanding process states is essential for debugging. The state field in `task_struct` tells you exactly what the kernel thinks a process is doing at any moment. This is the information visible in the `ps` command's `STAT` column.

| State | Macro | Wakeable by signal? | `ps` letter | Example cause |
|---|---|---|---|---|
| Running or runnable | `TASK_RUNNING` | — | R | On CPU or on a run queue |
| Interruptible sleep | `TASK_INTERRUPTIBLE` | Yes | S | Waiting for I/O, event, or timer |
| Uninterruptible sleep | `TASK_UNINTERRUPTIBLE` | No | D | DMA wait, kernel I/O path (V4L2, NVMe) |
| Killable | `TASK_KILLABLE` | SIGKILL only | D | Uninterruptible but yields to kill |
| Stopped | `__TASK_STOPPED` | SIGCONT | T | SIGSTOP or debugger attach |
| Zombie | `EXIT_ZOMBIE` | — | Z | Exited; awaiting parent `wait()` call |
| Dead | `TASK_DEAD` | — | — | Fully reclaimed after parent reaps |

`D` state in `ps` indicates a process blocked inside a kernel I/O path. Persistent `D` state is a driver hang indicator — common during V4L2 buffer dequeue failures or NVMe timeout.

```
Process State Machine
                   ┌───────────────────────────────┐
                   │                               │
           schedule()                      preempted / slice expires
                   │                               │
                   ▼                               │
           ┌─────────────┐     blocks on I/O   ┌──┴──────────┐
  fork() → │ TASK_RUNNING│ ─────────────────→  │ TASK_INTER- │
  exec()   │  (runnable) │ ←─────────────────  │  RUPTIBLE   │
           └──────┬──────┘    signal / event   └─────────────┘
                  │                                    │
        kernel DMA/I/O path                      SIGKILL only
                  │                                    ▼
                  ▼                          ┌──────────────────┐
           ┌────────────┐                   │  TASK_KILLABLE   │
           │  TASK_UN-  │                   └──────────────────┘
           │INTERRUPTIBLE│
           └──────┬──────┘
                  │
               exit()
                  ▼
           ┌────────────┐    parent wait()   ┌──────────┐
           │EXIT_ZOMBIE │ ─────────────────→ │TASK_DEAD │
           └────────────┘                   └──────────┘
```

> **Key Insight:** `TASK_UNINTERRUPTIBLE` exists because some kernel operations — particularly DMA transfers and hardware I/O — cannot be safely interrupted mid-way. If a process waiting on `VIDIOC_DQBUF` (V4L2 dequeue buffer) could be killed at any point, the DMA engine might write into freed memory. The `D` state is the kernel saying "I'm in the middle of a hardware operation; please wait." A persistent `D` state means the hardware never completed its operation.

> **Common Pitfall:** A zombie process (`Z` in `ps`) is not a bug in the child — it is a bug in the parent. The child has exited and freed its memory, but the kernel keeps a minimal `task_struct` entry until the parent calls `wait()` to collect the exit code. If a parent process never calls `wait()`, zombies accumulate and eventually exhaust the PID namespace. In openpilot, process supervisors must reap all child processes.

---

## fork / exec / wait

The fork/exec/wait trio is the fundamental mechanism for creating new processes in Unix. Understanding this sequence is also key to understanding why openpilot's multi-process architecture works efficiently.

### fork() and Copy-on-Write

`fork()` creates a child as a structural copy of the parent. Physical memory is **not** copied immediately — **Copy-on-Write** defers allocation:

```
fork() — Copy-on-Write Memory Model
┌─────────────┐   fork()   ┌─────────────┐
│   Parent    │ ─────────> │    Child    │
│   Process   │            │   Process   │
│             │            │             │
│ page table: │            │ page table: │
│  0x1000 ──────────────────────> [RO]  │  ← same physical page
│  0x2000 ──────────────────────> [RO]  │    marked read-only
│  0x3000 ──────────────────────> [RO]  │
└─────────────┘            └─────────────┘
                                  │
                           write to 0x2000
                                  │
                                  ▼
                           ┌─────────────┐
                           │ PAGE FAULT  │
                           │ kernel      │
                           │ allocates   │
                           │ new page    │
                           │ child 0x2000│
                           │ → new page  │
                           └─────────────┘
```

The sequence when `fork()` is called:

1. **Kernel copies `task_struct`**: a new `task_struct` is allocated and populated from the parent's, with a new PID.
2. **`mm_struct` is duplicated**: the new task gets its own virtual address space descriptor, but the page table entries point to the same physical pages as the parent.
3. **Pages marked read-only**: the kernel marks all shared pages read-only in both parent and child page tables.
4. **Child returns 0, parent returns child PID**: both resume execution from the instruction after `fork()`.
5. **On first write**: a page fault fires. The kernel allocates a new physical page, copies the content, and updates only the writing task's page table. This is the actual "copy" — deferred until necessary.
6. **Code pages are never copied**: read-only text segments (the program's executable code) are genuinely shared forever, never duplicated.

CoW makes `fork()` fast even for large processes. openpilot's multi-process architecture relies on this: `camerad`, `modeld`, `plannerd`, and `controlsd` each fork from a common base without duplicating megabytes of shared library code.

### exec() and wait()

`execve()` replaces the current address space with a new ELF binary. File descriptors without `O_CLOEXEC` survive across exec. `waitpid()` reaps a zombie child, reclaiming its `task_struct`. Without `wait()`, zombies accumulate and eventually exhaust PID space.

If a parent exits before reaping, orphan children are reparented to PID 1 (systemd), which calls `wait()` internally.

```
The fork / exec / wait lifecycle
┌─────────┐
│ Parent  │
│ Process │
└────┬────┘
     │ fork()
     ├──────────────────────────────────┐
     │                                  ▼
     │                           ┌─────────────┐
     │ (continues running)       │    Child    │
     │                           │  (PID = N)  │
     │                           └──────┬──────┘
     │                                  │ execve("/usr/bin/camerad")
     │                                  ▼
     │                           ┌─────────────┐
     │                           │  camerad    │
     │                           │  (new ELF)  │
     │                           └──────┬──────┘
     │                                  │ exit(0)
     │                                  ▼
     │                           ┌─────────────┐
     │ waitpid(N, &status, 0) ←─ │   ZOMBIE    │
     │ (reaps child)             │  (PID = N)  │
     ▼                           └─────────────┘
┌─────────┐
│ Parent  │
│(continues│
└─────────┘
```

---

## clone() and Threads

`clone()` is the underlying syscall behind both `fork()` and `pthread_create()`. The flags argument determines what the new task shares with its parent.

| Flag | Effect |
|---|---|
| `CLONE_VM` | Share `mm_struct` — both tasks use the same address space (thread) |
| `CLONE_FILES` | Share open file descriptor table |
| `CLONE_SIGHAND` | Share signal handlers |
| `CLONE_NEWPID` | New PID namespace — child is PID 1 inside it |
| `CLONE_NEWNET` | New network namespace — isolated interface/routing table |
| `CLONE_NEWNS` | New mount namespace — isolated filesystem view |

A thread is simply a task created with `CLONE_VM | CLONE_FILES | CLONE_SIGHAND`. `getpid()` returns `tgid` (same for all threads in a process); `gettid()` returns the unique per-thread `pid`.

> **Key Insight:** The fact that threads and processes are the same structure (`task_struct`) means the scheduler treats them identically. A thread at `SCHED_FIFO` priority 80 will preempt a process at priority 50 just as readily as it preempts another thread at priority 50. CPU affinity, cgroup membership, and scheduling class are per-`task_struct` — meaning you can set different scheduling policies for different threads within the same process.

---

## Linux Namespaces

**Namespaces** partition kernel resources so a set of processes sees an isolated view. They are the foundation of containers.

| Namespace | Isolates | Container use |
|---|---|---|
| `pid` | Process ID numbering | Container init appears as PID 1 |
| `mnt` | Filesystem mount tree | Container-private root filesystem |
| `net` | Network interfaces, routes, iptables | Per-container networking |
| `uts` | Hostname and domain name | Container-specific hostname |
| `ipc` | System V IPC, POSIX message queues | IPC isolation between containers |
| `user` | UID/GID mappings | Rootless containers |
| `cgroup` | cgroup root view | Nested cgroup hierarchies |
| `time` | Clock offsets | Time namespace per container |

```bash
ls -la /proc/[pid]/ns/          # inspect namespace membership of a running process
unshare --pid --fork bash       # launch shell in new PID namespace
```

GPU device files (`/dev/nvidia0`, `/dev/nvhost-ctrl`) must be bind-mounted into the container's mount namespace for CUDA to initialize inside containers.

> **Common Pitfall:** When running TensorRT or CUDA inside a Docker container, the container has its own mount namespace. The NVIDIA runtime must bind-mount `/dev/nvidia*` and `/dev/nvhost-*` into the container. If this fails silently, CUDA will report "no devices found" even though the host can see the GPU. Always check `docker run --gpus all` or the NVIDIA container runtime configuration before chasing a CUDA driver bug.

Now that we understand how processes are created and isolated, let's look at how the kernel limits what resources they can consume — cgroups.

---

## cgroups v2: Resource Control

**Unified hierarchy** at `/sys/fs/cgroup/`. All controllers (cpu, memory, io, cpuset) attach to the same hierarchy tree.

| Controller | Key file | Example value | Effect |
|---|---|---|---|
| cpu | `cpu.max` | `50000 100000` | 50% of one CPU (quota µs / period µs) |
| cpuset | `cpuset.cpus` | `0-3` | Restrict to cores 0–3 |
| cpuset | `cpuset.mems` | `0` | Restrict to NUMA node 0 |
| memory | `memory.max` | `4G` | OOM-kill if exceeded |
| memory | `memory.swap.max` | `0` | Disable swap for this group |
| io | `io.max` | `8:0 rbps=104857600` | 100 MB/s read on device 8:0 |
| pids | `pids.max` | `512` | Limit fork bombs in untrusted containers |

```bash
cat /proc/[pid]/cgroup              # cgroup membership path for a process
cat /sys/fs/cgroup/[path]/cpu.stat  # throttled_usec, nr_throttled — detect throttling
```

Kubernetes uses cgroup v2 to enforce CPU and memory limits on inference pods. A pod with `cpu.max = 200000 1000000` (20% of one core) will have `modeld` throttled if it exceeds that budget.

> **Key Insight:** `cpu.stat`'s `throttled_usec` field is the smoking gun for cgroup-induced latency. If your inference pod shows consistent 2–3 ms latency spikes and `throttled_usec` is climbing, the Kubernetes CPU limit is the bottleneck — not the model, not the GPU, not the scheduler. This is the first file to check after `perf` and `bpftrace` show CPU stalls in the inference thread.

> **Common Pitfall:** Setting `cpuset.cpus` without also setting `cpuset.mems` on a NUMA system can lead to memory being allocated from the wrong NUMA node. This causes cross-node memory traffic that adds ~100 ns per cache line miss. Always pair CPU affinity with NUMA memory node pinning for latency-sensitive inference workloads on multi-socket servers.

---

## Context Switch Mechanics

Now that we understand how the kernel tracks tasks and their resources, let's look at the operation that switches execution between them — the **context switch**.

`context_switch()` is in `kernel/sched/core.c`. It performs two distinct operations:

1. **`switch_mm_irqs_off()`** — install the new process's virtual address space. On x86 this writes CR3 (the page table base register); on ARM64 it writes TTBR0_EL1. This is the step that makes the new process's memory visible and hides the old process's memory. Every memory access after this point goes through the new page table.

2. **`switch_to()`** — save the outgoing task's callee-saved registers (rbx, rbp, r12–r15 on x86; x19–x28, fp, lr on ARM64) and stack pointer to its `task_struct`, then restore the incoming task's saved registers. When `switch_to()` returns, the CPU is executing in the context of the new task.

3. **Resume** — the new task resumes at the exact instruction where it was last preempted, as if nothing happened. Its register state, stack, and virtual memory are all restored.

```
Context Switch Timeline
┌──────────────┐                    ┌──────────────┐
│  Task A      │                    │  Task B      │
│  (running)   │                    │  (waiting)   │
└──────┬───────┘                    └──────┬───────┘
       │                                   │
       │ scheduler tick / block            │
       │                                   │
       ▼                                   │
┌─────────────────────────────────┐        │
│  context_switch(A → B)          │        │
│  1. switch_mm: write TTBR0/CR3  │        │
│  2. switch_to: save A's regs    │        │
│               restore B's regs  │        │
└─────────────────────┬───────────┘        │
                      │                    │
                      └───────────────────►│
                                           │  (Task B resumes here)
                                           ▼
                                    ┌──────────────┐
                                    │  Task B      │
                                    │  (running)   │
                                    └──────────────┘
```

**TLB cost**: ARM64 uses ASID-tagged TLBs — switching between tasks with valid ASIDs avoids a full TLB flush. x86 uses PCID for the same purpose. Context switch overhead: 1–10 µs depending on cache state and whether the TLB must be flushed. For a 1 kHz control loop (`controlsd` at 100 Hz CAN output), scheduler jitter must stay well below 1 ms.

> **Key Insight:** The TLB (Translation Lookaside Buffer) is a hardware cache that stores recent virtual-to-physical address translations. Without ASID tags, every context switch would require flushing the TLB entirely — that is, invalidating all cached translations — because the new process has a completely different address space. ASID tags let the hardware distinguish "translation for process A" from "translation for process B," so old entries remain valid and the new process can hit the TLB immediately. This is why ASID exhaustion (when all 256 or 65536 ASID slots fill up) forces a TLB flush and adds latency.

---

## /proc/[pid]/ Runtime Inspection

| Path | Contents |
|---|---|
| `/proc/[pid]/maps` | Virtual memory regions: address, permissions, backing file |
| `/proc/[pid]/smaps` | Per-region RSS and PSS; identifies memory waste and sharing |
| `/proc/[pid]/status` | State, VmRSS, threads, capability sets |
| `/proc/[pid]/fd/` | Symlinks to open files, sockets, V4L2 device nodes |
| `/proc/[pid]/sched` | CFS/EEVDF: vruntime, nr_voluntary_switches, se.load.weight |
| `/proc/[pid]/wchan` | Kernel function where task is currently sleeping |
| `/proc/[pid]/cgroup` | cgroup v2 membership path |
| `/proc/[pid]/oom_score` | OOM killer score; higher value killed first under memory pressure |
| `/proc/[pid]/oom_score_adj` | Writable: tune OOM priority (-1000 = never kill, +1000 = kill first) |

> **Common Pitfall:** Under memory pressure, the OOM killer selects the process with the highest `oom_score` to terminate. By default, large-memory processes score highest. On a Jetson running both `modeld` and a data logging service, the OOM killer may terminate `modeld` rather than the logger if `modeld` has a larger RSS. Set `oom_score_adj = -500` on critical inference processes to protect them. Conversely, set `oom_score_adj = +500` on non-critical logging processes so they are killed first.

---

## Summary

| State | Macro | Wakeable? | Example cause |
|---|---|---|---|
| Running / runnable | `TASK_RUNNING` | — | On CPU or waiting on run queue |
| Interruptible sleep | `TASK_INTERRUPTIBLE` | Yes (signal) | Blocked on `read()`, `epoll_wait()` |
| Uninterruptible sleep | `TASK_UNINTERRUPTIBLE` | No | DMA wait, `VIDIOC_DQBUF` in driver |
| Killable | `TASK_KILLABLE` | SIGKILL only | NFS soft mount wait |
| Stopped | `__TASK_STOPPED` | SIGCONT | Debugger, SIGSTOP |
| Zombie | `EXIT_ZOMBIE` | — | Awaiting parent `waitpid()` |

### Conceptual Review

- **Why does Linux use a single `task_struct` for both processes and threads?** Simplicity and consistency. The scheduler, OOM killer, cgroup accounting, and CPU affinity mechanisms all operate on `task_struct` without needing special cases for threads. The distinction between process and thread is entirely in which fields are shared (`mm`, `files`) via the `clone()` flags.
- **What is a zombie process and why does it exist?** A zombie is a process that has called `exit()` but whose parent has not yet called `wait()`. The kernel keeps a minimal `task_struct` so the parent can retrieve the child's exit status. Zombies consume a PID slot but no CPU or memory. They accumulate when a parent fails to reap its children.
- **Why does `fork()` use Copy-on-Write instead of immediately copying memory?** Copying the entire address space at `fork()` time would be prohibitively slow for large processes. Most `fork()+exec()` pairs never write to the parent's pages at all — `execve()` replaces the address space immediately. CoW defers the copy cost to the moment it is actually needed.
- **What does `TASK_UNINTERRUPTIBLE` mean in practice?** The process is blocked inside a kernel code path (typically a hardware I/O operation) that cannot be safely interrupted. You cannot kill a process in this state with SIGKILL — only when the kernel I/O path completes (or times out) will the process become killable. Persistent `D` state means the hardware is hung.
- **How does `clone()` relate to `fork()` and `pthread_create()`?** Both `fork()` and `pthread_create()` are implemented in terms of the `clone()` syscall. `fork()` calls `clone()` with no sharing flags (new `mm`, new `files`). `pthread_create()` calls `clone()` with `CLONE_VM | CLONE_FILES | CLONE_SIGHAND` (shared address space, shared file descriptors, shared signal handlers).
- **What is CPU affinity and why does it matter for inference?** `task_struct.cpus_mask` is a bitmask of CPUs the task is allowed to run on. Pinning `modeld` to the big cluster (e.g., Cortex-A78AE on Orin) prevents the scheduler from migrating it to a LITTLE core mid-inference. Migration causes cache invalidation and pipeline stalls; affinity eliminates this variability.

---

## AI Hardware Connection

- `task_struct.sched_class` determines the scheduler for each task; assigning `modeld` to `SCHED_FIFO` switches it to `rt_sched_class`, preventing CFS/EEVDF jitter from delaying frame processing by up to 5 ms on an untuned system.
- `cpuset.cpus` in cgroup v2 pins inference processes to isolated cores, preventing migration to cores shared with interrupt handlers; on Jetson Orin, the big-cluster Cortex-A78AE cores are typically reserved for `modeld` and `camerad`.
- `cpu.max` throttles background processes (telemetry, logging) to a fixed quota so inference threads retain burst CPU headroom — directly writable at `/sys/fs/cgroup/[pod]/cpu.max` in Kubernetes.
- openpilot's `camerad`, `modeld`, `sensord`, `plannerd`, and `controlsd` run as separate processes; CoW fork semantics give each an independent `mm_struct`, enabling crash isolation without corrupting sibling address spaces.
- `TASK_UNINTERRUPTIBLE` appears in camera and DMA driver code paths — a persistent `D`-state process in `/proc/[pid]/wchan` pointing to `v4l2_dqbuf` or `nvdla_submit` immediately identifies the stalled hardware interface.
- PID namespaces in Kubernetes inference pods isolate service process trees; `/proc/[pid]/cgroup` on the host maps any guest PID to its pod's resource accounting group for OOM investigation.
