3D Generation Tutorial En

§0 TL;DR Cheat Sheet

1. Three representations: NeRF (implicit neural field + volume rendering), 3DGS (explicit Gaussian point cloud + rasterization), Mesh / SDF (explicit surface / implicit distance field). Sweet spot for quality vs speed: 3DGS (Kerbl 2023 SIGGRAPH Best Paper).
2. NeRF core equation: $C(\mathbf{r}) = \int_{t_n}^{t_f} T(t)\sigma(\mathbf{r}(t))\mathbf{c}(\mathbf{r}(t),\mathbf{d})\,dt$, where $T(t) = \exp\!\left(-\int_{t_n}^{t}\sigma(\mathbf{r}(s))\,ds\right)$ is the transmittance. Discretization yields $\alpha$-compositing: $C \approx \sum_i T_i (1-e^{-\sigma_i\delta_i})\mathbf{c}_i$.
3. Instant-NGP (Müller 2022 SIGGRAPH): multi-resolution hash grid + tiny MLP, 5+ OOM speedup; hash collisions are automatically disambiguated by the MLP over colliding entries (suppressed jointly by the loss and multi-scale redundancy).
4. 3DGS core: scene represented as a set of 3D Gaussians $\{\mu_i, \Sigma_i, \alpha_i, c_i(\mathbf{d})\}$, differentiable rasterization projects the 3D covariance to 2D via Jacobian $J$: $\Sigma' = J W \Sigma W^\top J^\top$, then performs front-to-back alpha-blending after depth sorting.
5. DreamFusion SDS (Poole et al. 2022 arXiv → ICLR 2023 Outstanding): supervise a 3D representation using a pretrained 2D diffusion: $\nabla_\theta \mathcal{L}_\text{SDS} = \mathbb{E}_{t,\epsilon}[w(t)(\epsilon_\phi(x_t;y,t)-\epsilon)\,\partial x/\partial \theta]$, deliberately dropping the U-Net Jacobian of $x_t$ to make training simulation-free. Price: mode-seeking → over-saturation / Janus.
6. VSD (Wang 2023 NeurIPS, ProlificDreamer): treats the 3D parameters $\theta$ as a random variable $\mu(\theta)$ and minimizes the KL between the noised rendered image distributions: $\mathbb{E}_t\big[D_\text{KL}\big(q_\mu^t(x_t|y)\,\|\,p_\phi^t(x_t|y)\big)\big]$. The gradient form is a relative score $\nabla_\theta \approx (\epsilon_\phi(x_t;y,t) - \epsilon_\psi(x_t;y,t,\pi))\,\partial x/\partial\theta$, where $\epsilon_\psi$ is a LoRA-finetuned auxiliary score. CFG can drop from 100 to 7.5.
7. Single-image / Few-view 3D: Zero-1-to-3 (Liu 2023 ICCV) uses viewpoint-conditioned diffusion; SyncDreamer / MVDream learn joint multi-view consistency; TripoSR / InstantMesh / Stable Fast 3D push image-to-mesh to ≤3 seconds.
8. 3D Foundation Models (2024-25 open source): Trellis (Microsoft 2024) uses structured latent + flow matching; Hunyuan3D-2 (Tencent 2025) is shape→texture two-stage; CLAY (Zhang 2024 SIGGRAPH) uses large-scale latent diffusion + 3DShape2VecSet.
9. Embodied AI key applications: Sim2Real asset generation, NeRF/3DGS as differentiable simulators, language-conditioned 3D affordance. Common interview crossovers: NeRF SLAM, Gaussian-Splat scene editing, 3D physics consistency.

§1 Intuitive comparison of the three representations

The first multiple-choice question in 3D generation is representation — pick the wrong one and the entire downstream pipeline is wasted.

§2 NeRF: derivation of volume rendering (must-know)

2.1　Continuous volume-rendering equation

NeRF (Mildenhall 2020 ECCV Best Paper Honorable Mention) represents a scene as a 5D neural field $f_\theta : (\mathbf{x}, \mathbf{d}) \to (\sigma, \mathbf{c})$:

• Input: 3D position $\mathbf{x} \in \mathbb{R}^3$ + view direction $\mathbf{d} \in \mathbb{S}^2$
• Output: volume density $\sigma \ge 0$ (direction-independent) + color $\mathbf{c} \in \mathbb{R}^3$ (direction-dependent, captures specular reflection)

For camera ray $\mathbf{r}(t) = \mathbf{o} + t\mathbf{d}$, integrate along $t \in [t_n, t_f]$ to get pixel color:

$$\boxed{\;C(\mathbf{r}) = \int_{t_n}^{t_f} T(t)\,\sigma(\mathbf{r}(t))\,\mathbf{c}(\mathbf{r}(t),\mathbf{d})\,dt\;}$$

where transmittance (the probability that the ray has not been blocked between $t_n$ and $t$) is

$$\boxed{\;T(t) = \exp\!\left(-\int_{t_n}^{t}\sigma(\mathbf{r}(s))\,ds\right)\;}$$

2.2　Why this form? — derivation from physics

Consider a light ray traveling through a participating medium. Over $[t, t+dt]$:

• Probability of absorption / scattering out of the ray: $\sigma(\mathbf{r}(t))\,dt$
• Color contribution emitted by the medium at this point: $\mathbf{c}(\mathbf{r}(t),\mathbf{d})$

Let $T(t)$ be the survival probability of the ray from $t_n$ to $t$. From $t \to t + dt$, the survival probability changes as

$$T(t+dt) = T(t)\,(1 - \sigma\,dt) \;\Rightarrow\; \frac{dT}{dt} = -\sigma(t)\,T(t)$$

This is a first-order ODE with initial value $T(t_n) = 1$, solving to

$$T(t) = \exp\!\left(-\int_{t_n}^{t}\sigma(\mathbf{r}(s))\,ds\right)$$

The color contribution to the pixel at each depth $t$ = survival probability × absorption probability × local color:

$$dC = T(t)\,\sigma(t)\,\mathbf{c}(t)\,dt$$

Integrating gives $C(\mathbf{r})$.

2.3　Discretization: $\alpha$-compositing (must-derive)

Continuous integration is impossible in practice. Slice $[t_n, t_f]$ into $N$ segments with $\delta_i = t_{i+1} - t_i$, and assume $\sigma, \mathbf{c}$ are constants $\sigma_i, \mathbf{c}_i$ within each segment.

Within-segment transmittance decay: on $[t_i, t_{i+1}]$, $T$ satisfies $dT/dt = -\sigma_i T$, so

$$\frac{T(t_{i+1})}{T(t_i)} = e^{-\sigma_i \delta_i}$$

This gives inter-segment transmittance:

$$T_i := T(t_i) = \prod_{j=1}^{i-1} e^{-\sigma_j \delta_j} = \exp\!\Big(\!-\!\sum_{j=1}^{i-1}\sigma_j\delta_j\Big)$$

Within-segment color contribution (integral, not a simple rectangle):

$$\int_{t_i}^{t_{i+1}} T(t)\sigma_i\mathbf{c}_i\,dt = T_i\,\mathbf{c}_i \int_0^{\delta_i} \sigma_i e^{-\sigma_i s}\,ds = T_i\,\mathbf{c}_i\,(1 - e^{-\sigma_i\delta_i})$$

Let $\alpha_i := 1 - e^{-\sigma_i \delta_i}$ (segment opacity). Combining yields the discrete NeRF equation:

$$\boxed{\;C(\mathbf{r}) \approx \sum_{i=1}^{N} T_i\,\alpha_i\,\mathbf{c}_i,\quad T_i = \prod_{j

This is exactly the graphics front-to-back alpha-compositing formula. Key point: $\alpha_i = 1 - e^{-\sigma_i\delta_i}$, not $\sigma_i\delta_i$; they are approximately equal when $\sigma_i\delta_i$ is small (first-order Taylor), but differ noticeably when large (saturating to 1 vs diverging linearly).

2.4　Positional encoding $\gamma(p)$: representing high-frequency detail

MLPs default to a low-frequency bias (NTK analysis); fitting $f(\mathbf{x})$ directly produces blur. NeRF uses positional encoding to lift the frequency:

$$\gamma(p) = \big(\sin(2^0\pi p),\cos(2^0\pi p),\sin(2^1\pi p),\cos(2^1\pi p),\dots,\sin(2^{L-1}\pi p),\cos(2^{L-1}\pi p)\big)$$

Use $L=10$ for $\mathbf{x}$ (60 dims) and $L=4$ for $\mathbf{d}$ (24 dims). Tancik et al. 2020 "Fourier Features" later gave an NTK explanation: high-frequency $\sin/\cos$ slow the kernel decay, allowing the MLP to learn high frequencies.

2.5　Hierarchical sampling: coarse → fine

• Coarse network: uniformly sample $N_c = 64$ points, render to get weights $w_i = T_i \alpha_i$
• Fine network: normalize $w$ to a PDF and importance-sample $N_f = 128$ new points (importance sampling: regions near the surface have larger weights and should be densely sampled)
• Composite the final color with all $N_c + N_f$ coarse + fine points
• Loss: $\mathcal{L} = \|C_c - C_\text{gt}\|^2 + \|C_f - C_\text{gt}\|^2$ (supervise both networks; the coarse one provides a well-defined gradient for the sampler)

2.6　NeRF training code (core 30 lines)

import torch
import torch.nn as nn
import torch.nn.functional as F

def positional_encoding(x: torch.Tensor, L: int) -> torch.Tensor:
    """ x: [..., D]; returns [..., D*2*L]; NeRF γ(p) does not include the raw x """
    freqs = 2.0 ** torch.arange(L, device=x.device, dtype=x.dtype) * torch.pi
    args = x.unsqueeze(-1) * freqs                # [..., D, L]
    pe = torch.stack([torch.sin(args), torch.cos(args)], dim=-1)  # [..., D, L, 2]
    return pe.flatten(-3)                          # [..., D*2*L]

def volume_render(sigma: torch.Tensor, color: torch.Tensor, t_vals: torch.Tensor,
                  ray_d: torch.Tensor):
    """
    NeRF discrete α-compositing
        sigma:   [B, N]        volume density (>= 0; usually after softplus / ReLU)
        color:   [B, N, 3]     color
        t_vals:  [B, N]        t values of sampled points along the ray (monotonically increasing)
        ray_d:   [B, 3]        ray direction (used to convert Δt → true distance)
    Returns C: [B, 3], weights: [B, N], depth: [B]
    """
    # δ_i = t_{i+1} - t_i; pad the last segment with 1e10 (absorbs the remainder out to infinity)
    deltas = t_vals[..., 1:] - t_vals[..., :-1]
    delta_far = torch.full_like(deltas[..., :1], 1e10)
    deltas = torch.cat([deltas, delta_far], dim=-1)            # [B, N]
    deltas = deltas * torch.norm(ray_d[:, None, :], dim=-1)    # convert t-spacing to real Euclidean distance

    alpha = 1.0 - torch.exp(-sigma * deltas)                   # [B, N]
    # T_i = ∏_{j<i} (1 - α_j) — use cumprod, shifted by one so T_1 = 1
    T = torch.cumprod(torch.cat([torch.ones_like(alpha[..., :1]),
                                 1.0 - alpha + 1e-10], dim=-1), dim=-1)[..., :-1]
    weights = T * alpha                                        # [B, N]
    C = (weights[..., None] * color).sum(dim=-2)               # [B, 3]
    depth = (weights * t_vals).sum(dim=-1)                     # [B]
    return C, weights, depth

2.7　Mip-NeRF / Mip-NeRF 360 (anti-aliasing)

Vanilla NeRF aliases badly at low resolution / when zoomed (the same pixel corresponds to cones of different scales, but NeRF treats them as rays).

• Mip-NeRF (Barron 2021 ICCV): treats the ray as a cone (view frustum), uses Integrated Positional Encoding (IPE) — closed-form Gaussian expectation of PE over the cone segment $\mathbb{E}_{\mathbf{x}\sim\mathcal{N}(\mu,\Sigma)}[\gamma(\mathbf{x})]$. For frequency $\omega = 2^k\pi$, $\mathbb{E}[\sin\omega x] = \sin(\omega\mu)\,e^{-\frac{1}{2}\omega^\top\Sigma\,\omega}$; high-frequency coefficients are automatically attenuated by the cone covariance $\Sigma$ via $e^{-\frac{1}{2}\omega^\top\Sigma\omega}$, naturally achieving multi-scale behavior.
• Mip-NeRF 360 (Barron 2022 CVPR): for unbounded scenes, applies contraction $f(x) = (2 - 1/\|x\|)\,x/\|x\|$ for $\|x\| > 1$, squashing infinity into a ball; adds distortion / proposal MLP losses.

2.8　NeuS / VolSDF: volume rendering + SDF (key to mesh extraction)

NeRF is density-based; mesh extraction requires choosing a $\sigma$ threshold (unstable). NeuS (Wang 2021 NeurIPS) replaces density with SDF $d(\mathbf{x})$:

$$\sigma(t) = \max\!\left(\frac{-\frac{d}{dt}\Phi_s(d(\mathbf{r}(t)))}{\Phi_s(d(\mathbf{r}(t)))},\; 0\right),\quad \Phi_s(d) = (1 + e^{-sd})^{-1}$$

where $\Phi_s$ is the sigmoid and $s$ is a learnable "sharpness". Properties: the weight peaks at the surface ($d=0$); Marching Cubes can directly extract the mesh (the mesh is $\{d = 0\}$).

VolSDF (Yariv 2021 NeurIPS) uses Laplace CDF $\sigma = \alpha\,\Phi(-d/\beta)$, with a similar idea.

§3 Instant-NGP: 5+ OOM speedup (must-know)

Training a vanilla NeRF on one scene takes 1-2 days. Instant-NGP (Müller 2022 SIGGRAPH Best Paper) fits a simple scene in 5 seconds.

3.1　Core idea: multi-resolution hash grid

Replace "dense grid vs big MLP" with "sparse hash grid + tiny MLP".

• $L$ resolution levels (e.g. $L = 16$); the $\ell$-th level has $N_\ell = \lfloor N_\min \cdot b^\ell \rfloor$ grid points, geometric progression ($b \approx 1.38$; typical paper values $N_\min = 16$, $N_\max \in [512, 2048]$ depending on scene size)
• Each level uses a hash function to map grid-point coordinates into a fixed-size feature table ($T = 2^{14}$–$2^{24}$, typically $T = 2^{19} = 524288$)
• Query point $\mathbf{x}$: trilinear-interpolate at the 8 corner points per level → concatenate to a $L \times F$-dim feature ($F = 2$)
• Feed into a tiny MLP (2 layers, hidden 64) to output $\sigma, \mathbf{c}$

3.2　Hash function

$$\text{hash}(\mathbf{x}) = \bigg(\bigoplus_{i=1}^{d} x_i \cdot \pi_i\bigg) \bmod T$$

$\pi_i$ are large primes ($\pi_1 = 1, \pi_2 = 2654435761, \pi_3 = 805459861$). $\oplus$ is XOR. This is a spatial hash, commonly used in physics-simulation BVHs.

3.3　How are hash collisions disambiguated? (L3 high-frequency follow-up)

When $N_\ell^d > T$ (inevitable at fine levels), multiple grid points map to the same entry → collision. Why does it still work?

1. Multi-resolution redundancy: features at coarse levels are unique ($N_\ell^d \le T$); fine levels add detail. The MLP can recover structure from coarse features, with fine levels only responsible for detail.
2. Sparsity prevails: most of space is empty (most voxels in a NeRF scene are background), and meaningful queries concentrate near surfaces, so "valid colliding grid-point pairs" are rare.
3. Gradients auto-disambiguate: during training only grid points near the surface have non-zero gradients (weighted by ray weights). "Colliding entries" in empty regions receive no gradient signal and do not contaminate surface entries.
4. MLP post-processing: the tiny MLP learns a classification/regression on the $L \times F$ concatenated features; surface points with collisions can be disambiguated via non-colliding features from other levels.

3.4　Instant-NGP training equations

Parameters: hash table $\theta_\text{hash} \in \mathbb{R}^{L \times T \times F}$ + MLP weights $\theta_\text{MLP}$. The loss is still photometric MSE, but the 5-seconds-vs-1-day gap comes from:

• Tiny MLP: 100× fewer parameters, ~50× faster forward
• Hash table: sparse activations, cache-friendly
• Fused CUDA kernels: tiny-cuda-nn fuses forward + backward
• Occupancy grid: a coarse occupancy grid skips sampling in empty regions, avoiding wasted queries

3.5　Plenoxels / TensoRF (contemporary explicit methods)

Plenoxels (Fridovich-Keil 2022 CVPR): pure voxel grid + spherical harmonics SH coefficients + density, no MLP at all, directly gradient-descend on voxels; speed similar to Instant-NGP but heavy on VRAM. TensoRF (Chen 2022 ECCV): compresses the 4D tensor field via VM / CP decomposition, reducing parameters from $O(N^4)$ to $O(N)$ or $O(N^2)$.

§4 3D Gaussian Splatting: explicit differentiable rasterization (current workhorse)

3DGS (Kerbl 2023 SIGGRAPH Best Paper) addressed NeRF's two big pain points: slow rendering and hard editing.

4.1　Scene representation

Scene = a set of 3D Gaussians $\{G_i\}$, each:

• Mean $\mu_i \in \mathbb{R}^3$ (position)
• Covariance $\Sigma_i \in \mathbb{R}^{3\times 3}$ (shape), decomposed as $\Sigma = R S S^\top R^\top$ (rotation $R$ + diagonal scaling $S$)
• Opacity $\alpha_i \in [0, 1]$
• Color $c_i(\mathbf{d})$ via SH coefficients ($\ell = 3$, 16 coefficients per channel, 48 parameters total)

Why $R S S R^\top$ and not learn $\Sigma$ directly? — to ensure $\Sigma$ is positive definite. Learning $\Sigma$ as a raw matrix under gradient descent escapes the positive-semidefinite cone; the decomposition only requires $R$ to be orthogonal (parameterized via quaternion $q$) and $S$ to be positive (parameterized via $\exp(s)$), so it is satisfied automatically.

4.2　3D → 2D projection Jacobian (L3 must-derive)

To splat a 3D Gaussian to the screen for rasterization, the 3D covariance $\Sigma$ must be projected to a 2D covariance $\Sigma'$.

Step 1: World → Camera: rigid transform $W \in SE(3)$. $\Sigma_\text{cam} = W \Sigma W^\top$ (where $W$ takes the rotation part; translation does not affect covariance).

Step 2: Camera → Screen: perspective projection is nonlinear:

$$\pi(\mathbf{x}) = \begin{pmatrix} f_x\,x/z \\ f_y\,y/z \end{pmatrix}$$

The covariance of a nonlinear map is approximated via first-order Taylor. Compute the Jacobian at the mean $\mu_\text{cam} = (x, y, z)$:

$$J = \frac{\partial \pi}{\partial \mathbf{x}}\bigg|_{\mu_\text{cam}} = \begin{pmatrix}\dfrac{f_x}{z} & 0 & -\dfrac{f_x\,x}{z^2}\\[2pt] 0 & \dfrac{f_y}{z} & -\dfrac{f_y\,y}{z^2}\end{pmatrix} \in \mathbb{R}^{2\times 3}$$

Step 3: 2D covariance (core formula):

$$\boxed{\;\Sigma' = J\,W\,\Sigma\,W^\top\,J^\top \in \mathbb{R}^{2\times 2}\;}$$

Derivation: if $\mathbf{x} \sim \mathcal{N}(\mu, \Sigma)$, then first-order $\pi(\mathbf{x}) \approx \pi(\mu) + J(\mathbf{x} - \mu)$, so $\text{Cov}[\pi(\mathbf{x})] \approx J\,\Sigma_\text{cam}\,J^\top = J W \Sigma W^\top J^\top$. This is the classic corollary of EWA splatting (Zwicker 2001).

4.3　Differentiable rasterization: tile-based front-to-back alpha-blending

The color at pixel $\mathbf{p}$:

$$C(\mathbf{p}) = \sum_{i \in \mathcal{N}(\mathbf{p})}\,c_i\,\alpha_i\,G_i'(\mathbf{p}) \prod_{j < i}\big(1 - \alpha_j\,G_j'(\mathbf{p})\big)$$

where $G_i'(\mathbf{p}) = \exp\!\big(-\tfrac{1}{2}(\mathbf{p} - \mu_i')^\top \Sigma_i'^{-1} (\mathbf{p} - \mu_i')\big)$ is the value of the 2D Gaussian at the pixel, and $\mathcal{N}(\mathbf{p})$ are the Gaussians covering $\mathbf{p}$ sorted by depth.

Key engineering:

1. Tile partitioning: split the screen into $16\times 16$ tiles, sort Gaussians by depth within each tile, render in parallel
2. GPU sort: radix sort, with composite key (tile_id, depth)
3. Front-to-back early stop: exit when accumulated $\prod(1 - \alpha G') < 10^{-4}$
4. CUDA kernel: the authors release diff-gaussian-rasterization; both forward and backward are manual derivatives

4.4　3DGS forward pass (PyTorch reference implementation)

def quat_to_rot(q: torch.Tensor) -> torch.Tensor:
    """ q: [N, 4] (w, x, y, z) already normalized; returns R: [N, 3, 3] """
    w, x, y, z = q.unbind(-1)
    R = torch.stack([
        1 - 2*(y*y + z*z),   2*(x*y - w*z),     2*(x*z + w*y),
        2*(x*y + w*z),       1 - 2*(x*x + z*z), 2*(y*z - w*x),
        2*(x*z - w*y),       2*(y*z + w*x),     1 - 2*(x*x + y*y),
    ], dim=-1).reshape(-1, 3, 3)
    return R

def gaussian_splat_forward(
    means3D: torch.Tensor,        # [N, 3]  Gaussian centers (world)
    scales: torch.Tensor,          # [N, 3]  log-scale (take exp for true scale)
    quats: torch.Tensor,           # [N, 4]  quaternion (will be normalized)
    opacities: torch.Tensor,       # [N, 1]  σ(logit) → α
    colors: torch.Tensor,          # [N, 3]  (simplified to RGB here, no SH expansion)
    viewmat: torch.Tensor,         # [4, 4]  world→camera
    K: torch.Tensor,               # [3, 3]  intrinsics (fx, fy, cx, cy)
    H: int, W: int,
):
    """ Pedagogical forward: no tile sort / CUDA, just shows the math.
        Real production uses gsplat / diff-gaussian-rasterization. """
    N = means3D.shape[0]
    device = means3D.device

    # --- 1. World → Camera ---
    homo = torch.cat([means3D, torch.ones(N, 1, device=device)], dim=-1)
    mu_cam = (homo @ viewmat.T)[:, :3]                          # [N, 3]
    z = mu_cam[:, 2].clamp(min=1e-4)                            # guard against div-by-zero

    # --- 2. Covariance (3D) ---
    q = quats / quats.norm(dim=-1, keepdim=True)
    R = quat_to_rot(q)                                          # [N, 3, 3]
    S = torch.diag_embed(torch.exp(scales))                     # [N, 3, 3]
    cov3D = R @ S @ S.transpose(-1, -2) @ R.transpose(-1, -2)   # [N, 3, 3]

    # Apply the World→Cam rotation part W_rot (3x3) to the covariance
    W_rot = viewmat[:3, :3]
    cov_cam = W_rot @ cov3D @ W_rot.T                           # [N, 3, 3]

    # --- 3. Projection Jacobian J (2x3) ---
    fx, fy = K[0, 0], K[1, 1]
    x_c, y_c, z_c = mu_cam[:, 0], mu_cam[:, 1], z
    J = torch.zeros(N, 2, 3, device=device)
    J[:, 0, 0] = fx / z_c
    J[:, 0, 2] = -fx * x_c / z_c**2
    J[:, 1, 1] = fy / z_c
    J[:, 1, 2] = -fy * y_c / z_c**2

    # --- 4. 2D covariance Σ' = J W Σ W^T J^T ---
    cov2D = J @ cov_cam @ J.transpose(-1, -2)                   # [N, 2, 2]
    cov2D = cov2D + 0.3 * torch.eye(2, device=device)           # low-pass filter (anti-aliasing)

    # --- 5. 2D center (pixel coordinates) ---
    cx, cy = K[0, 2], K[1, 2]
    mu2D = torch.stack([fx * x_c / z_c + cx, fy * y_c / z_c + cy], dim=-1)  # [N, 2]

    # --- 6. Depth sort (front to back) ---
    depth = z_c
    order = depth.argsort()                                     # ascending z
    mu2D, cov2D = mu2D[order], cov2D[order]
    colors_o = colors[order]
    alphas = torch.sigmoid(opacities[order]).squeeze(-1)        # [N]

    # --- 7. Pixel loop (pedagogical full-image loop; real impl uses tile + CUDA) ---
    yy, xx = torch.meshgrid(torch.arange(H, device=device),
                            torch.arange(W, device=device), indexing='ij')
    pix = torch.stack([xx, yy], dim=-1).float()                 # [H, W, 2]

    img = torch.zeros(H, W, 3, device=device)
    T_acc = torch.ones(H, W, device=device)
    inv_cov = torch.linalg.inv(cov2D)                           # [N, 2, 2]

    for i in range(mu2D.shape[0]):
        diff = pix - mu2D[i]                                    # [H, W, 2]
        # 2D Gaussian: exp(-0.5 (diff)^T Σ^-1 (diff))
        G = torch.exp(-0.5 * (diff @ inv_cov[i] * diff).sum(-1))  # [H, W]
        contrib = alphas[i] * G                                 # [H, W]
        contrib = contrib.clamp(max=0.99)                       # numerical safety
        img = img + (T_acc * contrib).unsqueeze(-1) * colors_o[i]
        T_acc = T_acc * (1 - contrib)
        if (T_acc < 1e-4).all():                                # early stop
            break

    return img

4.5　Adaptive density control (frequent interview topic)

3DGS initializes from sparse SfM (COLMAP) point clouds, but training must "spread" more Gaussians.

Heuristic conditions: gradient norm > τ_pos (e.g. $2 \times 10^{-4}$), scale > τ_scale (1% of scene scale).

def densify_and_prune(gaussians, grad_thresh=2e-4, scale_thresh=0.01,
                      max_screen_size=None):
    """ Pedagogical densify decisions (simplified; real gsplat also has screen-size triggers).
        Assume gaussians exposes the following 1D fields (N = current Gaussian count):
          xyz_grad_accum: [N]  ‖accumulated xyz gradient norm‖
          denom:          [N]  accumulation count (avoid /0)
          scales:         [N, 3]  log-scale
          opacities:      [N]  ∈ (0, 1) after sigmoid
          screen_size:    [N]  last-render screen-projected size (optional)
          grad_dir:       [N, 3]  last gradient direction (for clone offset)
    """
    grad_norm  = gaussians.xyz_grad_accum / gaussians.denom.clamp(min=1)      # [N]
    mean_scale = gaussians.scales.exp().max(dim=-1).values                    # [N]

    # CLONE: high gradient + small scale — duplicate and offset along the gradient; keep original
    clone_mask = (grad_norm > grad_thresh) & (mean_scale <= scale_thresh)     # [N]
    gaussians.clone_at(clone_mask, offset=gaussians.grad_dir[clone_mask])

    # SPLIT: high gradient + large scale — split into 2 children (scale ÷ 1.6), remove original at the end
    split_mask = (grad_norm > grad_thresh) & (mean_scale >  scale_thresh)     # [N]
    gaussians.split_at(split_mask, n=2, scale_div=1.6)

    # PRUNE: low opacity / too large on screen / already marked by split
    # ⚠️ Note: new Gaussians from clone are appended to the end; length has changed. The mask only applies to the original N.
    prune_mask = (gaussians.opacities[:split_mask.shape[0]] < 0.005) | split_mask
    if max_screen_size is not None:
        prune_mask = prune_mask | (gaussians.screen_size[:split_mask.shape[0]] > max_screen_size)
    gaussians.remove_original(prune_mask)   # delete only the originals flagged among the first N

    gaussians.reset_grad_accum()
    return gaussians

4.6　2DGS / Surfels (surface-aligned)

3DGS ellipsoids are not surface-aware; mesh extraction needs SuGaR / GSDF post-processing. 2DGS (Huang 2024 SIGGRAPH) degenerates the 3D ellipsoid into a 2D disk (one axis = 0), directly aligning with the surface, which works better with normal/depth supervision and produces clearly higher-quality meshes.

4.7　Dynamic 4DGS

Dynamic 3DGS (Luiten 2024 3DV) uses independent per-frame Gaussians + physics prior to link them; 4DGS (Wu 2024 CVPR / Yang 2024 ICLR) writes $\mu(t), \Sigma(t)$ as functions of time (MLP or spline); SC-GS (Huang 2024) drives dense Gaussians from sparse control points (analogous to LBS).

§5 Mesh extraction: Marching Cubes / DMTet

After NeRF / 3DGS reconstruction, downstream (simulator, AR, 3D printing) often needs a mesh.

5.1　Marching Cubes (classic must-know)

Input: 3D scalar field $f(\mathbf{x})$ (density / SDF) + threshold $\tau$. Output: triangle mesh of the level set $\{f = \tau\}$.

Algorithm skeleton:

1. Voxelize space (8 corners per voxel)
2. For each voxel, binarize the 8 corners ($f > \tau$ as 1, else 0) → 256 possible configurations
3. Look up table: each configuration has predefined isosurface triangle patches + vertex positions on edges
4. Linear interpolation for precise vertices: between edge endpoints $\mathbf{a}, \mathbf{b}$, interpolate $t = (\tau - f(\mathbf{a})) / (f(\mathbf{b}) - f(\mathbf{a}))$, vertex $= \mathbf{a} + t(\mathbf{b} - \mathbf{a})$
5. Merge all voxel triangles → full mesh

def marching_cubes_sketch(density: torch.Tensor, threshold: float):
    """ Real implementations use mcubes / scikit-image / pytorch3d; this shows the idea """
    from skimage.measure import marching_cubes
    # density: [Nx, Ny, Nz]; detach() cuts the autograd graph, cpu().numpy() moves to host
    verts, faces, normals, _ = marching_cubes(
        density.detach().cpu().numpy(),
        level=threshold,
        spacing=(1.0, 1.0, 1.0),
        gradient_direction='descent',  # normal direction; descent = surface faces low density
    )
    return verts, faces, normals

5.2　Differentiable: DMTet / FlexiCubes (end-to-end mesh learning)

Marching Cubes is not differentiable (the lookup table is discrete).

• DMTet (Shen 2021 NeurIPS / Munkberg 2022 CVPR): uses a deformable tetrahedral grid, each tetrahedron has 4 vertex SDFs + position offsets that are differentiable. Marching Tetrahedra replaces MC; topology is determined by SDF signs and geometry by vertex positions, fully differentiable.
• FlexiCubes (Shen 2023 SIGGRAPH): generalizes dual marching cubes by introducing extra learnable parameters (dual vertex offsets / interpolation weights) to fix quality artifacts.

Typical use: Magic3D and Fantasia3D after DreamFusion use DMTet to learn meshes + textures under SDS supervision.

§6 SDS Loss: supervising 3D with 2D diffusion (DreamFusion family)

6.1　Problem setup

We want to generate 3D assets but have no 3D training data — 3D data is scarce (ShapeNet ~50k objects, Objaverse-XL 10M but uneven quality). Pretrained 2D diffusion (Stable Diffusion, Imagen) is abundant. Can we use 2D diffusion as a teacher to supervise 3D?

DreamFusion (Poole et al. 2022 arXiv → ICLR 2023 Outstanding Paper) proposed Score Distillation Sampling (SDS).

6.2　Setup

• 3D representation $\theta$ (NeRF parameters / DMTet vertices / 3DGS point cloud)
• Differentiable renderer $g(\theta, \pi) \to x$, where $x$ is an image ($\pi$ is the camera viewpoint)
• Pretrained 2D diffusion $\epsilon_\phi(x_t; y, t)$ ($y$ is the text prompt)

Goal: make $g(\theta, \pi)$ look like a "photo of $y$", i.e. $g(\theta, \pi)$ lies on the data manifold learned by the diffusion model.

6.3　SDS gradient derivation (L3 must-know)

Intuition: backprop the diffusion training loss into $\theta$. Naive idea: treat the rendered image $x = g(\theta, \pi)$ as a training sample and minimize

$$\mathcal{L}_\text{diff}(\theta) = \mathbb{E}_{t, \epsilon}\Big[w(t)\big\|\epsilon_\phi(x_t; y, t) - \epsilon\big\|^2\Big],\quad x_t = \alpha_t x + \sigma_t \epsilon$$

Take the gradient w.r.t. $\theta$ (chain rule):

$$\nabla_\theta \mathcal{L}_\text{diff} = \mathbb{E}\Big[w(t)\,2\big(\epsilon_\phi(x_t; y, t) - \epsilon\big)\,\underbrace{\frac{\partial \epsilon_\phi(x_t;y,t)}{\partial x_t}}_{\text{U-Net Jacobian}}\,\alpha_t\,\underbrace{\frac{\partial x}{\partial \theta}}_{\text{renderer Jacobian}}\Big]$$

Problem: the U-Net Jacobian $\partial \epsilon_\phi / \partial x_t$ is expensive to compute and numerically poor (the diffusion model is large and not trained for second-order stability).

SDS trick: drop the U-Net Jacobian entirely, giving

$$\boxed{\;\nabla_\theta \mathcal{L}_\text{SDS} \;=\; \mathbb{E}_{t, \epsilon}\Big[w(t)\,\big(\epsilon_\phi(x_t; y, t) - \epsilon\big)\,\frac{\partial x}{\partial \theta}\Big]\;}$$

(The original DreamFusion paper writes it as $\partial L/\partial \theta$; $\alpha_t$ and the constant 2 are absorbed into $w(t)$.)

6.4　Why does dropping the Jacobian still work?

First explanation (original DreamFusion, score view): $\epsilon_\phi(x_t; y, t)/\sigma_t \approx -\nabla_{x_t}\log p_\phi(x_t|y)$ (score). The SDS gradient = (predicted score - noise) × renderer Jacobian, i.e. it pushes the rendered image toward high-probability regions.

Second explanation (mode-seeking): SDS is equivalent to a mode-seeking form of $\mathbb{E}_t[D_\text{KL}(q(x_t|\theta) \,\|\, p_\phi(x_t|y))]$: drive toward high-probability regions of $p_\phi(\cdot|y)$.

6.5　SDS side effects: over-saturation / mode collapse / Janus

• Over-saturation: oversaturated colors, excessive contrast ("plastic-y look")
• Over-smoothing: blurred details
• Mode collapse: objects converge to "canonical" single forms
• Janus problem: 3D objects grow a "front face" in every view (faces on the back of heads; animals with heads on both sides)

Root cause: SDS is equivalent to a mode-seeking KL, and only large CFG (DreamFusion default 100) can escape the mean-mode. CFG=100 sharpens the distribution to the extreme → over-saturation.

6.6　SDS code (core 30 lines)

def sds_loss(
    renderer,                 # θ → x (B, 3, H, W)
    theta,                    # 3D parameters (NeRF / 3DGS / DMTet)
    prompt_emb,               # text embedding (cond) [B, L, D]
    uncond_emb,               # text embedding (uncond / null) [B, L, D]
    unet,                     # frozen 2D diffusion U-Net (e.g. SD); returns noise pred Tensor
    alpha_cumprod: torch.Tensor,  # [T_max] precomputed bar-alpha schedule
    cfg_scale: float = 100.0,
    t_range: tuple = (0.02, 0.98),
):
    """ Score Distillation Sampling loss (DreamFusion).
        Convention: unet(x_t, t, encoder_hidden_states=emb) -> [B, 3, H, W] noise pred.
        If using diffusers UNet2DConditionModel, wrap to take .sample.
        Returns a grad surrogate, can be backward'd directly. """
    x = renderer(theta)                                  # [B, 3, H, W]
    B = x.shape[0]
    device, dtype = x.device, x.dtype
    T_max = alpha_cumprod.shape[0]                       # usually 1000

    # 1. Sample t and noise, forward add noise
    t = torch.randint(int(t_range[0] * T_max), int(t_range[1] * T_max),
                      (B,), device=device)
    noise = torch.randn_like(x)
    abar = alpha_cumprod.to(device=device, dtype=dtype)[t].view(B, 1, 1, 1)
    x_t = abar.sqrt() * x + (1 - abar).sqrt() * noise

    # 2. U-Net predicts noise (cond / uncond), CFG combination; key: no autograd through diffusion
    with torch.no_grad():
        eps_uncond = unet(x_t, t, encoder_hidden_states=uncond_emb)
        eps_cond   = unet(x_t, t, encoder_hidden_states=prompt_emb)
        eps_pred = eps_uncond + cfg_scale * (eps_cond - eps_uncond)

    # 3. SDS gradient: w(t)(ε_pred - ε) · ∂x/∂θ; w(t) = σ_t² is a common choice
    grad = ((1 - abar) * (eps_pred - noise)).detach()
    # backward of (grad · x) gives grad · ∂x/∂θ
    return (grad * x).sum() / B

6.7　VSD: variational SDS (ProlificDreamer, NeurIPS 2023 Spotlight)

VSD (Wang 2023 NeurIPS) views SDS as a special case of "point estimation for a single $\theta$" and generalizes to variational inference over a distribution $\mu(\theta)$ on $\theta$.

Setup

• Treat the 3D parameters $\theta$ as a latent random variable with distribution $\mu(\theta)$
• Goal: align the rendered-image distribution with the diffusion-learned prior distribution (not mode alignment)

Objective and gradient

ProlificDreamer writes the objective as a KL:

$$\min_{\mu}\; D_\text{KL}\!\Big(q_\mu^t(x_t|y)\;\Big\|\;p_\phi^t(x_t|y)\Big),\quad t\sim\mathcal{U}[0,1]$$

where $q_\mu^t$ is the distribution induced by "rendering from $\theta\sim\mu$ + adding noise to time $t$". The variational gradient (Wang et al. 2023, Theorem 2, abbreviated) gives the update direction in $\theta$ as a relative score:

$$\boxed{\;\nabla_\theta \mathcal{L}_\text{VSD} \;=\; \mathbb{E}_{t,\epsilon}\Big[\,w(t)\,\big(\epsilon_\phi(x_t;y,t) \;-\; \epsilon_\psi(x_t;y,t,\pi)\big)\,\frac{\partial x}{\partial \theta}\,\Big]\;}$$

Compared to SDS: replace the raw noise $\epsilon$ with an auxiliary score $\epsilon_\psi$. $\epsilon_\psi$ is a LoRA-finetuned score network that online-minimizes a score-matching loss to track the score of the current $q_\mu^t$; it plays the role of a "variance-reduction baseline" (isomorphic to a value baseline in RL actor-critic). Note: the above is the gradient form, not a squared-loss form; the paper does not have an implementable form "write a scalar loss $\|\epsilon_\phi-\epsilon_\psi\|^2$ and then differentiate" — $\epsilon_\psi$ depends on $\mu$, which would drop the key term of the KL.

Intuition for why over-saturation is mitigated

• SDS: pushes the rendered $x$ toward modes of the prior $p_\phi(\cdot|y)$ (mean-mode → needs CFG=100 → over-saturation)
• VSD: the auxiliary score tracks the score of the current rendered distribution; the update direction = from "where I am now" to "where the prior is" (relative gradient), without needing huge CFG. CFG can drop to 7.5 (the standard diffusion default), avoiding extreme sharpening
• Empirically: VSD has more natural colors and more complex geometry, and can maintain multiple modes simultaneously (ProlificDreamer reports 50k-step training yielding photorealistic Buddha statues, etc.)

6.8　SDS derivative family: mesh / 3DGS + SDS

§7 Single-image / Few-view 3D generation

A more practical setting: given one image, generate 3D.

7.1　Zero-1-to-3 paradigm (novel view via diffusion)

Zero-1-to-3 (Liu 2023 ICCV): finetune Stable Diffusion on Objaverse so it accepts (input view, target camera) → output novel view.

• Input: single image $x$ + relative camera pose $\Delta R, \Delta T$
• Diffusion conditioning: image embedding (CLIP) + camera embedding (sinusoidal)
• Output: the image from the $\Delta R, \Delta T$ viewpoint

Usage: given one input view, sample 16-32 novel views, then reconstruct via NeRF / 3DGS.

Derivatives:

• Zero-1-to-3++ (Shi 2023): fixed generation of 6 anchor views (north-pole view + 4 eye-level views + a top view), reducing randomness
• SyncDreamer (Liu 2024 ICLR): joint prediction of multiple views in latent space (cross-attention lets views see each other), ensuring 3D consistency
• MVDream (Shi 2024 ICLR): text-to-multi-view, generates 4 views simultaneously; followed by SDS refinement

7.2　One-2-3-45 / InstantMesh / TripoSR / Stable Fast 3D

LRM (Hong et al. 2023 arXiv → ICLR 2024) setting: treat the image as tokens + Plucker ray embedding, transformer outputs a NeRF triplane. This is the parent model of TripoSR / InstantMesh.

7.3　LRM Triplane representation (high-frequency interview topic)

• Triplane (Chan 2022 EG3D): 3 axis-aligned 2D planes (XY, YZ, XZ), total $3 \times C \times N \times N$ dim
• Query 3D point $(x, y, z)$: bilinearly interpolate on each plane → concat → small MLP → $(\sigma, \mathbf{c})$
• Advantages: less VRAM than a voxel grid ($O(N^2)$ vs $O(N^3)$), denser than a hash grid making it suitable as transformer output
• LRM / TripoSR / InstantMesh all let the transformer directly regress triplane tokens

§8 3D Foundation Models (the 2024 open-source wave)

8.1　Trellis (Microsoft 2024, open source)

Trellis (Xiang 2024 arXiv) is the first attempt at "Stable Diffusion for 3D" in the open-source space.

• Structured Latent (SLAT): encode the 3D asset onto a sparse latent grid over voxels — preserving spatial structure (suitable for sparse conv / sparse attention) while being compact (only active voxels store latents)
• 3D VAE: mesh + texture (derived from signed distance field) → SLAT
• Flow matching prior: rectified flow on SLAT, conditioned on text/image
• Multiple decoders: decode SLAT into NeRF / 3DGS / mesh representations (same latent, choice of output format)
• Training data: a subset of Objaverse-XL + internal high-quality set
• Effects: text-to-3D / image-to-3D, seconds to tens of seconds, quality surpassing the SDS family

8.2　Hunyuan3D-1 / -2 (Tencent 2024-25, open source)

Hunyuan3D follows the shape-then-texture two-stage route.

• Hunyuan3D-1 (Yang 2024 arXiv):

  • Stage 1: text/image → multi-view image (Zero-1-to-3 family)
  • Stage 2: multi-view → 3D mesh (LRM-like reconstructor)
  • Outputs textured mesh in seconds to tens of seconds

• Hunyuan3D-2 (Tencent 2025, arXiv 2501.12202):

  • Hunyuan3D-DiT: geometry-only DiT generating mesh on SDF latents
  • Hunyuan3D-Paint: multi-view PBR texture diffusion, UV-space refinement
  • High-quality PBR texture (production-ready for game / VR assets)
• Open source: complete weights + inference code on HuggingFace

8.3　CLAY (Zhang 2024 SIGGRAPH)

• 3DShape2VecSet latent diffusion: represent the mesh as a vector set + cross-attention DiT
• Large-scale training (Objaverse-XL + internal curated set)
• Output SDF → marching cubes → mesh
• Adds a PBR texture stage (similar to Hunyuan3D-2)

Rodin (Microsoft 2023, commercial): early production-grade text-to-3D-avatar system, diffusion on triplane, focused on characters / avatars.

8.4　Comparison table

§9 Complexity / resource comparison

§10 Comparison with related methods & Embodied AI applications

10.1　Key differences between 3D and 2D generation

10.2　Embodied AI / AR / VR practical routes

§11 Engineering practice & common footguns

11.1　COLMAP / SfM preprocessing (essential for reconstruction)

Input multi-view → output intrinsics $K$ + extrinsics $\{R_i, t_i\}$ + sparse point cloud; standard pipeline SIFT → matching → incremental SfM → bundle adjustment. Common pitfalls: SfM fails on texture-less / specular objects; dynamic objects pollute extrinsics.

11.2　Numerical stability (general NeRF/3DGS)

11.3　Multi-machine distributed & evaluation metrics

Distributed: NeRF / Instant-NGP / 3DGS are single-GPU standard; large 3DGS scenes use chunk-wise (VastGaussian, CityGaussian); SDS/VSD runs 2 SD forwards per iter, 8×A100 gives significant speedup; Trellis / Hunyuan3D training is large-scale multi-node DDP.

§12 25 frequently-asked interview questions

Sorted into 3 tiers by difficulty (L1 must-know / L2 advanced / L3 top labs). Each question links to answer points + footguns.

L1 must-know (asked at every 3D / vision role)

L2 advanced (research-oriented roles)

L3 top-lab (top-conference / industry research roles)

§A Appendix: full code skeleton + references

A.1　Complete from-scratch code includes

volume_render() (NeRF α-compositing with numerical stability) · positional_encoding() (γ(p) Fourier features) · gaussian_splat_forward() (3DGS pedagogical forward + projection Jacobian) · densify_and_prune() (3DGS densification heuristics) · sds_loss() (SDS gradient surrogate) · marching_cubes_sketch() (mesh extraction interface using scikit-image).

A.2　Key papers reading list

• NeRF family: Mildenhall 2020 ECCV (HM); Müller Instant-NGP SIGGRAPH 2022 Best; Barron Mip-NeRF / 360 ICCV 2021 / CVPR 2022; Wang NeuS + Yariv VolSDF NeurIPS 2021; Fridovich-Keil Plenoxels CVPR 2022; Chen TensoRF ECCV 2022.
• 3DGS family: Kerbl 3D Gaussian Splatting SIGGRAPH 2023 Best; Huang 2D Gaussian Splatting SIGGRAPH 2024; Luiten Dynamic 3DGS 3DV 2024; Wu 4DGS CVPR 2024; Guédon SuGaR CVPR 2024.
• Mesh / SDF: Shen DMTet NeurIPS 2021 / FlexiCubes SIGGRAPH 2023.
• SDS family: Poole DreamFusion arXiv 2022.09 → ICLR 2023 Outstanding; Wang ProlificDreamer (VSD) NeurIPS 2023 Spotlight; Lin Magic3D CVPR 2023; Chen Fantasia3D ICCV 2023; Tang DreamGaussian ICLR 2024; Yi GaussianDreamer CVPR 2024.
• Single-image 3D: Liu Zero-1-to-3 ICCV 2023 / One-2-3-45 NeurIPS 2023 / SyncDreamer ICLR 2024; Shi Zero-1-to-3++ arXiv 2023 / MVDream ICLR 2024; Hong LRM arXiv 2023.11 → ICLR 2024; Tochilkin TripoSR arXiv 2024; Xu InstantMesh arXiv 2024; Boss Stable Fast 3D arXiv 2024.
• 3D Foundation Models: Xiang Trellis arXiv 2024 (Microsoft); Tencent Hunyuan3D-2 arXiv 2501.12202 (2025); Zhang CLAY SIGGRAPH 2024.

A.3　Common Embodied AI / AR / VR follow-ups

3DGS connected to a physics engine → first extract mesh via 2DGS / SuGaR → IsaacSim / MuJoCo; dynamic NeRF → 4DGS / D-NeRF / K-Planes; real-time AR 3DGS → mobile-friendly (PostShot, Luma) + pruning / quantization; insufficient 3D data → Objaverse-XL (Trellis), 2D distillation (DreamFusion family), or multi-view heuristics (MVDream).

3D Generation Quick Reference · Main references: Mildenhall 2020 (NeRF), Müller 2022 (Instant-NGP), Kerbl 2023 (3DGS), Poole 2022/ICLR 2023 (DreamFusion), Wang 2023 (VSD), Xiang 2024 (Trellis), Tencent 2025 (Hunyuan3D-2). Covers: NeRF volume-rendering derivation, Instant-NGP hash grid, 3DGS projection Jacobian, SDS / VSD gradient derivation, single-image 3D, 3D foundation models. Essential for Embodied AI / AR / VR.