Image Generation Systems Tutorial En

§0 TL;DR Cheat Sheet

1. LDM essentials: VAE encode compresses $H\times W\times 3$ to $h\times w\times c$ (SD 1.x: $8\times$ downsample, $c=4$); diffusion runs in latent space, saving $8^2=64\times$ compute, then VAE decode reconstructs pixels (Rombach et al. 2022 CVPR).
2. SD 1.x → SDXL → SD3 → FLUX lineage: 1.x uses CLIP-L text encoder + U-Net; SDXL has dual encoders (OpenCLIP-G + CLIP-L) + 2.6B U-Net + size/crop conditioning + Refiner (Podell et al. 2024 ICLR); SD3 switches to MM-DiT + Rectified Flow (Esser et al. 2024 ICML); FLUX.1 is a 12B MM-DiT with parallel attention (Black Forest Labs 2024).
3. CFG (must-know): at training, with probability $p_\text{drop}\approx 0.1$ replace condition $c$ with $\emptyset$; at inference, output $\hat\epsilon_\text{cfg} = \hat\epsilon_\emptyset + s\,(\hat\epsilon_c - \hat\epsilon_\emptyset)$, $s \in [1.5, 12]$ (Ho & Salimans 2022).
4. ControlNet zero-conv (Zhang et al. 2023 ICCV): deep-copy the entire U-Net encoder as a trainable copy; the $1\times 1$ conv connecting each branch to the backbone is initialized $W=0,b=0$ — forward pass is identity but gradient is nonzero ($\partial L/\partial W = \delta \cdot x \neq 0$). Training starts from a clean identity map and progressively injects condition signal, preserving pretrained capability.
5. IP-Adapter (Ye et al. 2023): Decoupled Cross-Attention — for image conditioning, add a new set of $W_K', W_V'$ in parallel with the text cross-attn; outputs are summed: $\text{out} = \text{Attn}(Q, K_\text{txt}, V_\text{txt}) + \lambda\,\text{Attn}(Q, K_\text{img}, V_\text{img})$. Only the new K/V + projector are trained, ~22M parameters.
6. LoRA (Hu et al. 2022 ICLR): $\Delta W = B A,\ B \in \mathbb{R}^{d\times r},\ A \in \mathbb{R}^{r\times k},\ r \ll \min(d,k)$; only $A, B$ are trained, $W$ is frozen. On SD, typical $r \in \{4,8,16,32\}$, 50–200× smaller than full fine-tune; at inference merge $W' = W + \alpha B A$.
7. DreamBooth (Ruiz et al. 2023 CVPR): rare-token (e.g., sks dog) + prior preservation loss $L = \|\epsilon - \hat\epsilon(x_t, t, \text{"a sks dog"})\|^2 + \lambda \|\epsilon' - \hat\epsilon(x_t', t, \text{"a dog"})\|^2$; the second term prevents language drift / overfitting.
8. DiT vs MM-DiT: DiT uses AdaLN-Zero (condition $\to$ MLP $\to$ scale/shift/gate, last layer's $W_\text{gate}=0$ for identity warm start, Peebles & Xie 2023 ICCV); MM-DiT concatenates text and image tokens into a single sequence for joint self-attention (each modality has independent QKV projections, but attention is global), enabling bidirectional information flow (Esser et al. 2024).

§1 Intuition and Big Picture

Why LDM? Pixel-space diffusion is too expensive: in the SD era, a 1024² image = 3M pixels, and a U-Net forward at full resolution per timestep means single-card training is restricted to toy 64×64 sizes. LDM's idea: use a pretrained VQ-VAE / KL-VAE to compress images to latents (SD 1.x: $512\times 512\times 3 \to 64\times 64\times 4$, $64\times$ fewer tokens), run diffusion only in latent space, then VAE decode back to pixels — decoupling "semantics / structure / texture":

    pixels x        latent z (perceptually similar)        latent z_T (noise)
   [H,W,3]    →     [h,w,c]     →     diffuse     →     [h,w,c]
                       ▲                                    │
                       │  VAE decoder                       │  reverse SDE / ODE
                       │                                    ▼
                    pixels x̂          ←      latent z_0      ←    [h,w,c]

The production stack decomposes into 5 orthogonal, swappable modules:

• Semantic conditioning (text): text encoder embedding → cross-attention K/V; primarily drives "what to draw"
• Structural conditioning (edges / depth / pose): ControlNet / T2I-Adapter; drives "what structure to follow"
• Identity / style conditioning (face / style): IP-Adapter / InstantID / PuLID / DreamBooth / LoRA; drives "whose look / whose style"

§2 LDM Core: VAE Compression + Latent Diffusion

2.1　LDM loss (Rombach et al. 2022 CVPR)

A pretrained KL-VAE gives encoder $\mathcal{E}: \mathbb{R}^{H\times W\times 3} \to \mathbb{R}^{h\times w\times c}$ and decoder $\mathcal{D}$ satisfying $\mathcal{D}(\mathcal{E}(x)) \approx x$ (perceptual reconstruction), with $h = H/f$ and downsample factor $f \in \{4, 8, 16\}$; the SD family uses $f=8$.

Training diffusion in latent space, the objective matches pixel-space DDPM:

$$\boxed{\;\mathcal{L}_\text{LDM} = \mathbb{E}_{z_0, \epsilon, t, c}\left[\,\big\|\epsilon - \epsilon_\theta\!\left(z_t,\, t,\, \tau_\theta(c)\right)\big\|^2\,\right]\;}$$

with $z_0 = \mathcal{E}(x)$, $z_t = \sqrt{\bar\alpha_t}\, z_0 + \sqrt{1-\bar\alpha_t}\,\epsilon$, and $\tau_\theta(c)$ the text encoder output.

2.2　Why $f=8$ is the sweet spot

Rombach 2022 Table 8 ablation:

Key insight: VAE reconstruction quality is an upper bound — no matter how strong the latent diffusion model is, it cannot produce images the VAE cannot decode. So larger $f$ is not always better; balance "compression ratio" vs "reconstruction ceiling".

2.3　VAE's "small KL" detail

The SD VAE is a KL-VAE, not a VQ-VAE — its latent is a continuous Gaussian with a tiny KL term (~$10^{-6}$ magnitude), effectively an AE with mild regularization. Rombach 2022 Appendix explains: too strong a KL would collapse the latent to a pure Gaussian, losing structural information.

§3 SD 1.x → SDXL → SD3 → FLUX Lineage

3.1　SD 1.x / 2.x (Stability AI 2022)

• U-Net 860M parameters: cross-attention is placed at three downsample levels of latent resolution 64 / 32 / 16 + corresponding upsample blocks + middle block (SD v1 config attention_resolutions = [4, 2, 1], DS=1/2/4 i.e. 64/32/16; the deepest DS=8 i.e. 8×8 downsample/upsample block only has ResBlocks without transformer, but the middle block at 8×8 does have transformer); text comes from CLIP-L (openai/clip-vit-large-patch14, 768-d), providing 77 token embeddings
• Objective: $\epsilon$-prediction (DDPM Ho et al. 2020 NeurIPS)
• Training resolution: 512², 1B+ LAION images
• 2.x switches to OpenCLIP-H/14 (stronger and cleaner license) and fine-tunes at 768²

3.2　SDXL (Podell et al. 2024 ICLR)

Three key upgrades:

3.3　SD3 (Esser et al. 2024 ICML)

Two core changes:

1) Training objective switches to Rectified Flow (RF)

$$x_t = (1-t)\, x_0 + t\, x_1,\quad u_t = x_1 - x_0$$

where $x_0 \sim \mathcal{N}(0, I)$ at the noise end, $x_1$ at the data end. The model learns $v_\theta(x_t, t, c) \approx x_1 - x_0$. Loss uses logit-normal $t$ sampling (middle $t$ has higher density) + RF weighting. Note the timestep convention is opposite to DDPM: the SD3 paper uses $t=0$ at noise, $t=1$ at data, but some code bases (diffusers) revert to the SD-style convention — proactively disambiguate in interviews.

2) Denoiser switches to MM-DiT (Multimodal Diffusion Transformer)

Text and image tokens are concatenated into a single sequence for joint self-attention; each modality has independent QKV projection + AdaLN-Zero MLP parameters, but the attention matrix is global (bidirectional). This means:

[txt tokens, img tokens]  ──╮
       │                     ├─ joint self-attention  ──→  bidirectional flow
       │                     │                            (txt sees img, img sees txt)
       │                     │
   independent QKV (txt, img)│
   independent LN/MLP gate (txt, img)

Compared to SD 1.x/SDXL cross-attention: image queries unidirectionally read text K/V, text is never updated. MM-DiT lets text be updated by image (opening the "image → text" flow); empirically text alignment improves noticeably.

3.4　FLUX.1 (Black Forest Labs 2024)

12B-parameter MM-DiT v2, with the main differences:

3.5　Parallel open-source lineage

• PixArt-α / Σ (Chen et al. 2024): DiT-XL/2 + T5 text, emphasizing "training cost only 12% of SDXL" — small but capable.
• Hunyuan-DiT (Tencent 2024 arXiv 2405.08748): Chinese-friendly bilingual DiT, 1.5B parameters, CLIP + mT5 dual encoders.
• DiT (Peebles & Xie 2023 ICCV): replaces the U-Net denoiser with a ViT-style transformer, class token + AdaLN-Zero conditioning; scaling laws smoother than U-Net — ancestor of SD3/FLUX.
• U-ViT (Bao et al. 2023): U-Net-style backbone but pure transformer blocks + long skip connections, an early transformer-based diffusion exploration.
• Imagen (Saharia et al. 2022 NeurIPS): Google's pixel-space (not latent) cascade — $64\times 64$ base + $256\times 256$ super-res + $1024\times 1024$ super-res; text uses a large T5-XXL, and results show text encoder scale > U-Net scale matters more for text alignment.

§4 DiT Architecture and AdaLN-Zero (Must-Know)

4.1　DiT block (Peebles & Xie 2023 ICCV)

DiT adapts the ViT block for diffusion: each block is conditioned by $c = \text{embed}(t) + \text{embed}(\text{class})$.

Input tokens x_l (shape [B, N, D]),  condition c (shape [B, D])
                │
        ┌───────┴────────┐
        │                │
   MLP(c) → (α₁, β₁, γ₁) │  scale / shift / gate parameters
        │                │
        ▼                │
   LayerNorm(x_l)        │
        │                │
   scale·γ₁ + shift·β₁   │  ← AdaLN: normalize then conditioned affine
        │                │
   Multi-Head Attention  │
        │                │
   × α₁ (gate, 0-init)   │  ← gate × residual; α₁ starts at 0
        │                │
   +  x_l                │  residual
        │                │
        ▼                │
   ┌────────────┐        │
   │ second half│        │
   │ (LN + MLP) │       same (α₂, β₂, γ₂) ← MLP(c)
   └────────────┘
        │
        ▼
   Output x_{l+1}

4.2　AdaLN-Zero derivation ("why is the gate initialized to 0")

DiT's actual form (Peebles & Xie 2023, Eqn. (5)–(6)): condition $c$ goes through a single MLP that produces $(\beta_1, \gamma_1, \alpha_1, \beta_2, \gamma_2, \alpha_2)$, and the normalization uses (1 + gamma) rather than gamma directly:

$$\text{AdaLN}(x, c) = \big(1 + \gamma(c)\big) \odot \text{LN}(x) + \beta(c)$$

AdaLN-Zero initializes the final weight + bias of the MLP that produces $(\beta, \gamma, \alpha)$ to 0:

$$\text{Block}(x, c) = x + \alpha(c) \cdot f\!\left(\big(1 + \gamma(c)\big) \odot \text{LN}(x) + \beta(c)\right)$$

At training step 0: MLP is all-zero → $\gamma = 0, \beta = 0, \alpha = 0$ → AdaLN degenerates to $\text{LN}(x)$, gate $\alpha = 0$ → block output $= x$ (identity). Note it's 1 + gamma, so gamma=0 doesn't zero out the normalization path; the normalized branch just equals LN(x).

$$\frac{\partial L}{\partial \alpha} = \frac{\partial L}{\partial \text{out}} \cdot f\!\left((1+\gamma)\odot\text{LN}(x) + \beta\right)$$

at step 0, $\gamma = \beta = 0$, and $f((1+0)\cdot\text{LN}(x) + 0) = f(\text{LN}(x))$ is nonzero (LN(x) is generally nonzero, and attention/MLP do not map arbitrary inputs to 0); thus $\partial L/\partial \alpha \neq 0$, and chain-ruling back to $W^{\text{last}}_\text{MLP}$ yields nonzero gradient — $\alpha$ grows from 0 and the block gradually forks a non-trivial transformation from the identity map, keeping training stable.

Note that $\gamma, \beta$ themselves have zero gradient at step 0 (their downstream is gated by $\alpha = 0$: $\partial L/\partial \gamma = (\partial L/\partial \alpha\cdot$ ...) — this path must pass through $\alpha$, and when $\alpha = 0$, $\partial \text{Block}/\partial \gamma$ contains an $\alpha\cdot f'(\cdot)$ factor equal to 0). But once $\alpha$ has grown, $\gamma, \beta$ immediately receive nonzero gradients — so "$\alpha$ grows first, then $\gamma, \beta$ follow" is the two-stage dynamics of AdaLN-Zero.

Compare naive initialization (standard random $\alpha \neq 0$): early blocks already produce large-variance outputs; stacked over 24-32 layers, activations explode and training diverges. AdaLN-Zero is the key design enabling DiT to scale.

4.3　Time embedding

$$\text{TimeEmbed}(t) = \text{MLP}\!\left(\text{SinusoidalEmb}(t)\right),\quad \text{SinusoidalEmb}(t)_{2i} = \sin\!\left(t / 10000^{2i/D}\right)$$

Even/odd positions use sin / cos, similar to Transformer positional encoding. $t$ is discrete timestep $\in \{0, 1, ..., T-1\}$ in SD; continuous $\in [0, 1]$ in RF / FM.

§5 SD Inference Loop + CFG (Core Code)

5.1　CFG formula

At training, with probability $p_\text{drop} \approx 0.1$, replace $c$ with null (empty text embedding or zero embedding) so a single network learns both conditional and unconditional branches. At inference:

$$\boxed{\;\hat\epsilon_\text{cfg}(z_t, t, c) = \hat\epsilon_\theta(z_t, t, \emptyset) + s\cdot\left[\hat\epsilon_\theta(z_t, t, c) - \hat\epsilon_\theta(z_t, t, \emptyset)\right]\;}$$

$s$ is the guidance scale; SD 1.x typically $s \in [5, 12]$; SDXL $s \approx 5$-$7$; FLUX dev $\approx 3.5$ (smaller, since RF models are more CFG-sensitive).

Under v-prediction / RF the form is identical, replacing $\hat\epsilon$ with $\hat v$.

5.2　SD inference loop (core 40 lines)

import torch

@torch.no_grad()
def sd_sample(unet, vae, text_encoder, tokenizer, scheduler,
              prompt, neg_prompt="", num_steps=30, cfg_scale=7.0,
              height=512, width=512, device="cuda", dtype=torch.float16):
    # 1) text encoding: forward both prompt and negative prompt
    ids_pos = tokenizer(prompt, padding="max_length", max_length=77,
                        truncation=True, return_tensors="pt").input_ids.to(device)
    ids_neg = tokenizer(neg_prompt, padding="max_length", max_length=77,
                        truncation=True, return_tensors="pt").input_ids.to(device)
    emb_pos = text_encoder(ids_pos)[0]                  # [1, 77, D_text]
    emb_neg = text_encoder(ids_neg)[0]
    emb = torch.cat([emb_neg, emb_pos], dim=0)          # [2, 77, D_text]  (uncond, cond)

    # 2) latent initialization: pure noise [1, 4, h/8, w/8]
    lat_shape = (1, 4, height // 8, width // 8)
    z = torch.randn(lat_shape, device=device, dtype=dtype) * scheduler.init_noise_sigma

    # 3) scheduler sets timesteps
    scheduler.set_timesteps(num_steps, device=device)

    # 4) main loop
    for t in scheduler.timesteps:
        # batch trick: pair the two latents with (uncond, cond) and forward once,
        # saves one kernel launch and is more cuDNN-batch-friendly than two sequential forwards
        z_in = torch.cat([z, z], dim=0)                  # [2, 4, h, w]
        z_in = scheduler.scale_model_input(z_in, t)      # some samplers need sigma scaling

        eps = unet(z_in, t, encoder_hidden_states=emb).sample   # [2, 4, h, w]
        eps_neg, eps_pos = eps.chunk(2, dim=0)

        # 5) CFG combine
        eps_cfg = eps_neg + cfg_scale * (eps_pos - eps_neg)

        # 6) scheduler step: invert eps to z_{t-1}
        z = scheduler.step(eps_cfg, t, z).prev_sample

    # 7) VAE decode + denormalize back to pixels [0, 1]
    z = z / 0.18215                                       # SD 1.x scaling factor
    x = vae.decode(z).sample                              # [1, 3, H, W] in [-1, 1]
    x = ((x.clamp(-1, 1) + 1) / 2)                        # → [0, 1]
    return x

§6 ControlNet and IP-Adapter: Conditioning Extensions

6.1　ControlNet architecture (Zhang et al. 2023 ICCV)

Problem: injecting edge / depth / pose and other structural conditions into a pretrained SD is too costly from scratch, and full fine-tuning destroys text-to-image capability.

Solution:

    Input latent z_t  ──┬───────────────────►  Original SD U-Net Encoder (frozen)
                        │                              │
                        │    Condition image c_img     │
                        │           │                  │
                        │           ▼                  │
                        │    Hint Encoder (conv stack) │
                        │           │                  │
                        │           ▼                  │
                        └────► Trainable Copy ←────────┤  encoder deep-copied,
                                    │                  │  starts training
                                    │                  │
                              Zero Conv (W=0, b=0)     │
                                    │                  │
                                    ▼                  │
                              add to skip connection ──┘  ─────►  Decoder (frozen)

• Trainable copy: full deep copy of SD U-Net's encoder + middle block as a sibling branch, initial weights = original SD encoder weights
• Zero-conv: each $1\times 1$ conv connecting back to the main skip path has weight and bias both initialized to 0
• Hint encoder: a 4-layer conv projecting the condition image (1 or 3 channels) into latent shape
• At training: original SD encoder/decoder are frozen; only trainable copy + zero-conv + hint encoder are trained

6.2　Zero-conv gradient derivation (L3 must-ask)

Zero-conv layer: $y = W \star x + b$, $W = 0, b = 0$, so $y = 0$; adding to the main skip equals "adding nothing", so forward is identity.

Backward splits into two paths:

(a) zero-conv's own weights:

$$\frac{\partial L}{\partial W_{ij}} = \frac{\partial L}{\partial y_i} \cdot x_j$$

$x_j$ (the zero-conv input, from the trainable copy output) is nonzero, $\partial L / \partial y_i$ is nonzero, so the gradient is nonzero — $W$ updates away from 0.

(b) trainable copy's parameters $\theta_c$: must pass through the $x \to y$ path; the chain rule's critical factor is $\partial y / \partial x = W$. At step 0 $W = 0$, so the trainable copy's own parameter gradient is 0 in the first step.

Hence convergence has two stages:

1. Step 0: $W = 0$ → ControlNet has zero effect on the backbone → output = original SD output → cannot be worse than baseline
2. Step 1: zero-conv breaks zero on its own (path (a)) → $W \neq 0$ → path (b) unlocks
3. Step 2+: trainable copy starts receiving gradients and learning
4. End of training: $W$ and the trainable copy together reach appropriate magnitudes; structural conditioning is integrated

6.3　Hint encoding + zero-conv (core 50 lines)

import torch
import torch.nn as nn

def zero_module(m: nn.Module) -> nn.Module:
    """ Zero out all parameters of a module (used for ControlNet zero-conv and IP-Adapter projector). """
    for p in m.parameters():
        nn.init.zeros_(p)
    return m

class HintEncoder(nn.Module):
    """ Encode the condition image (e.g. canny edge, depth) down to latent resolution. """
    def __init__(self, in_ch=3, out_ch=320):  # 320 = SD U-Net first hidden dim
        super().__init__()
        # progressively downsample to 1/8 (same factor as VAE); last layer is a zero-conv
        self.net = nn.Sequential(
            nn.Conv2d(in_ch, 16, 3, padding=1),       nn.SiLU(),
            nn.Conv2d(16, 16, 3, padding=1),          nn.SiLU(),
            nn.Conv2d(16, 32, 3, padding=1, stride=2), nn.SiLU(),   # /2
            nn.Conv2d(32, 32, 3, padding=1),          nn.SiLU(),
            nn.Conv2d(32, 96, 3, padding=1, stride=2), nn.SiLU(),   # /4
            nn.Conv2d(96, 96, 3, padding=1),          nn.SiLU(),
            nn.Conv2d(96, 256, 3, padding=1, stride=2), nn.SiLU(),  # /8
            zero_module(nn.Conv2d(256, out_ch, 3, padding=1)),
        )

    def forward(self, hint):     # hint: [B, 3, H, W]
        return self.net(hint)    # [B, out_ch, H/8, W/8]

class ControlNetBlock(nn.Module):
    """ trainable copy output → zero-conv → add to backbone skip """
    def __init__(self, ch):
        super().__init__()
        # this is the "output zero-conv" connecting to the backbone skip
        self.zero_conv = zero_module(nn.Conv2d(ch, ch, 1))   # 1×1, init 0

    def forward(self, x_copy, x_main_skip):
        # x_copy: current layer's output from the trainable copy
        # x_main_skip: original SD U-Net's skip activation at the same level
        return x_main_skip + self.zero_conv(x_copy)

6.4　T2I-Adapter (Mou et al. 2024) comparison

ControlNet's trainable copy is heavy (~half of SD's parameters). T2I-Adapter's idea is a pure adapter:

6.5　IP-Adapter (Ye et al. 2023)

IP-Adapter uses a reference image for conditioning, preserving identity / style across generations. The core is Decoupled Cross-Attention:

    image  ──► CLIP image encoder ──► [N_img, D_clip]
                                            │
                                            ▼
                                    Projector (Linear, ~22M params)
                                            │
                                            ▼
                                    image embeddings [N_img, D_text]
                                            │
                                            │  used in parallel with text embedding
                                            │
    text   ──► text encoder ──► [N_txt, D_text]
                │                                                          ▲
                │                                                          │
                ▼                                                          │
   ┌───────────────────────────────────────────────────────────────┐
   │   At each U-Net cross-attention layer:                         │
   │                                                                │
   │   Q = z W_Q                                                    │
   │                                                                │
   │   Original text path:   K_txt = c_txt W_K^txt,  V_txt = c_txt W_V^txt   │
   │   New image path:       K_img = c_img W_K^img, V_img = c_img W_V^img   │
   │                                                                │
   │   out = Attn(Q, K_txt, V_txt) + λ · Attn(Q, K_img, V_img)      │
   │                                                                │
   │   Only W_K^img, W_V^img, and projector are trained             │
   └───────────────────────────────────────────────────────────────┘

Total parameters ~22M (projector ~10M + new K/V projections per layer ~12M total), ~1% of full SD.

6.6　Decoupled Cross-Attention (core 45 lines)

import torch
import torch.nn as nn
import torch.nn.functional as F

class DecoupledCrossAttention(nn.Module):
    """ Parallel text + image cross-attention with summed outputs. """
    def __init__(self, d_model, num_heads, d_text, d_image, lam=1.0):
        super().__init__()
        self.h = num_heads
        self.d = d_model
        self.d_head = d_model // num_heads
        self.lam = lam

        # text path: same as original SD; load pretrained weights and **freeze**
        self.W_Q     = nn.Linear(d_model, d_model, bias=False)   # query from latent
        self.W_K_txt = nn.Linear(d_text,  d_model, bias=False)
        self.W_V_txt = nn.Linear(d_text,  d_model, bias=False)
        self.W_O     = nn.Linear(d_model, d_model, bias=False)
        for p in (*self.W_Q.parameters(),
                  *self.W_K_txt.parameters(),
                  *self.W_V_txt.parameters(),
                  *self.W_O.parameters()):
            p.requires_grad_(False)        # ← IP-Adapter only trains the new K/V

        # image path: new K/V; only these two + the projector are trainable
        self.W_K_img = nn.Linear(d_image, d_model, bias=False)
        self.W_V_img = nn.Linear(d_image, d_model, bias=False)

    def _split_heads(self, x):     # [B, L, D] → [B, H, L, d_head]
        B, L, _ = x.shape
        return x.view(B, L, self.h, self.d_head).transpose(1, 2)

    def _attn(self, Q, K, V):       # standard scaled dot-product
        return F.scaled_dot_product_attention(Q, K, V)   # [B, H, L_q, d_head]

    def forward(self, z, c_text, c_image):
        # z: [B, L_z, D]   c_text: [B, L_t, d_text]   c_image: [B, L_i, d_image]
        Q     = self._split_heads(self.W_Q(z))
        K_txt = self._split_heads(self.W_K_txt(c_text))
        V_txt = self._split_heads(self.W_V_txt(c_text))
        K_img = self._split_heads(self.W_K_img(c_image))
        V_img = self._split_heads(self.W_V_img(c_image))

        out_txt = self._attn(Q, K_txt, V_txt)
        out_img = self._attn(Q, K_img, V_img)
        out = out_txt + self.lam * out_img            # ← decoupled sum

        # [B, H, L_q, d_head] → [B, L_q, D]
        B, _, L_q, _ = out.shape
        out = out.transpose(1, 2).contiguous().view(B, L_q, self.d)
        return self.W_O(out)

6.7　InstantID / PuLID / PhotoMaker

The goal for all is single-reference-image identity-preserving text-to-image. Mainstream approaches:

Common thread: ID-relevant signal goes through dedicated adapters; ID-irrelevant signal (pose / expression / lighting) stays prompt-controlled, avoiding direct identity paste.

§7 Personalization: DreamBooth / Textual Inversion / LoRA / Custom Diffusion

7.1　Textual Inversion (Gal et al. 2023 ICLR)

Train only one token embedding, leave the model untouched:

1. Introduce a new token S* (e.g., <my-cat>); its embedding $e_{S^*} \in \mathbb{R}^{d_\text{text}}$ is the only trainable parameter
2. Training objective:

$$e_{S^*}^* = \arg\min_{e} \mathbb{E}_{z, \epsilon, t}\left[\|\epsilon - \epsilon_\theta(z_t, t, c(\text{"a photo of } S^*\text{"}; e))\|^2\right]$$

1. ~3-5K steps to converge; embedding is only 768-1024 dimensions, file size < 10KB

Pros: extremely lightweight, no catastrophic forgetting; Cons: limited expressivity (one embedding can't capture complex concepts).

7.2　DreamBooth (Ruiz et al. 2023 CVPR)

Two core ingredients:

1) Rare token + class word: use a rare token (e.g., sks, zwx) + class word (dog, person), with prompts shaped like "a photo of sks dog". Rare tokens have "semantic dead zones" in pretrained embeddings, free from interference by existing concepts.

2) Prior Preservation Loss:

$$\boxed{\;\mathcal{L} = \mathbb{E}\!\left[\|\epsilon - \hat\epsilon_\theta(z_t, t, c_\text{sks})\|^2\right] + \lambda\,\mathbb{E}\!\left[\|\epsilon' - \hat\epsilon_\theta(z_t', t, c_\text{class})\|^2\right]\;}$$

The second term is "class-prior preservation" — use the model's own generated class images (e.g., 200 images of "a photo of dog") as anchors, telling the model "sks dog is special, but a generic dog must still be drawn correctly". $\lambda$ is usually 1.0.

7.3　LoRA (Hu et al. 2022 ICLR)

Core math: instead of fine-tuning $W \in \mathbb{R}^{d \times k}$ directly, learn a low-rank delta:

$$\boxed{\;W' = W + \Delta W,\quad \Delta W = B A,\quad B \in \mathbb{R}^{d \times r},\ A \in \mathbb{R}^{r \times k},\ r \ll \min(d, k)\;}$$

Typically $A$ is initialized $\mathcal{N}(0, \sigma^2)$, $B$ is initialized to 0 → $\Delta W = 0$ (preserving pretrained behavior) → only $A, B$ are updated. At inference: $W' = W + \alpha B A$ ($\alpha$ is a scaling factor).

Parameter savings: $W$ has $d \cdot k$ params; LoRA has $r(d + k)$. On SD's U-Net cross-attn, one $W_K$ is $D \times d_\text{text}$ (e.g., $1280 \times 2048 = 2.6M$); with $r = 8$, LoRA is $8(1280 + 2048) = 26K$ — 100× fewer.

7.4　LoRA injection into nn.Linear (core 40 lines)

import torch
import torch.nn as nn

class LoRALinear(nn.Module):
    """ Wrap nn.Linear with low-rank delta. Original weight is frozen. """
    def __init__(self, base: nn.Linear, rank=8, alpha=8.0, dropout=0.0):
        super().__init__()
        self.base = base                          # freeze original Linear
        for p in self.base.parameters():
            p.requires_grad_(False)

        d_in, d_out = base.in_features, base.out_features
        self.rank, self.alpha = rank, alpha
        self.scale = alpha / rank                 # unified inference scaling

        # ΔW = B A,  A: [r, d_in],  B: [d_out, r]
        self.A = nn.Parameter(torch.empty(rank, d_in))
        self.B = nn.Parameter(torch.zeros(d_out, rank))   # B = 0  → ΔW = 0
        nn.init.kaiming_uniform_(self.A, a=5**0.5)        # like nn.Linear default

        self.drop = nn.Dropout(dropout) if dropout > 0 else nn.Identity()

    def forward(self, x):
        # original path: use frozen base
        out = self.base(x)
        # LoRA delta: x @ A^T @ B^T  (order matters — avoid the large [d_out, d_in] matrix)
        out = out + self.drop(x) @ self.A.t() @ self.B.t() * self.scale
        return out

def inject_lora(model, target_module_names=("to_q", "to_k", "to_v"),
                rank=8, alpha=8.0):
    """ Replace cross-attn / self-attn to_q/to_k/to_v Linears in SD U-Net with LoRALinear;
        to_out in Diffusers is nn.Sequential([Linear, Dropout]); handle to_out.0 separately. """
    replaced = 0
    for name, mod in model.named_modules():
        for child_name, child in list(mod.named_children()):
            # 1) to_q / to_k / to_v: single Linear, replace directly
            if (child_name in target_module_names
                    and isinstance(child, nn.Linear)):
                setattr(mod, child_name, LoRALinear(child, rank=rank, alpha=alpha))
                replaced += 1
            # 2) to_out: in Diffusers Attention, to_out is nn.Sequential(Linear, Dropout);
            #    SDXL LoRA conventionally wraps only the 0-th (Linear)
            if child_name == "to_out" and isinstance(child, nn.Sequential):
                lin = child[0]
                if isinstance(lin, nn.Linear):
                    child[0] = LoRALinear(lin, rank=rank, alpha=alpha)
                    replaced += 1
    return replaced

7.5　DreamBooth + LoRA = LoRA-DreamBooth

In practice pure DreamBooth is rare (full-train is too heavy); the mainstream is LoRA-DreamBooth: inject LoRA only into the Q/K/V/O of attention, with prior preservation. Files ~50-200MB, single-card 30 minutes, much more reproducible than pure DreamBooth.

7.6　DreamBooth training step (core 35 lines)

import torch
import torch.nn.functional as F

def dreambooth_train_step(unet, vae, text_encoder, scheduler,
                          x_instance, c_instance,    # training images + "a sks dog" embedding
                          x_class, c_class,          # self-generated class images + "a dog" embedding
                          lam_prior=1.0, dtype=torch.float16, device="cuda"):
    """ One DreamBooth + prior preservation training step """
    bs = x_instance.shape[0]

    # 1) concat instance & class into a 2× batch for a single forward
    x = torch.cat([x_instance, x_class], dim=0)
    c = torch.cat([c_instance, c_class], dim=0)        # text embeddings

    # 2) encode to latent + scale
    with torch.no_grad():
        z = vae.encode(x).latent_dist.sample() * 0.18215    # [2bs, 4, h, w]

    # 3) sample random timestep + noise
    t = torch.randint(0, scheduler.num_train_timesteps, (z.shape[0],), device=device)
    eps = torch.randn_like(z)
    z_t = scheduler.add_noise(z, eps, t)

    # 4) predict ε
    eps_pred = unet(z_t, t, encoder_hidden_states=c).sample

    # 5) split loss into instance / class halves
    eps_pred_inst, eps_pred_cls = eps_pred.chunk(2, dim=0)
    eps_inst, eps_cls = eps.chunk(2, dim=0)

    loss_inst = F.mse_loss(eps_pred_inst.float(), eps_inst.float(),
                           reduction="mean")
    loss_cls  = F.mse_loss(eps_pred_cls.float(),  eps_cls.float(),
                           reduction="mean")
    loss = loss_inst + lam_prior * loss_cls

    return loss

7.7　HyperDreamBooth / Custom Diffusion comparison

• HyperDreamBooth (Ruiz et al. 2024): use a hypernetwork to directly predict per-reference-image LoRA weights, enabling inference-time personalization (~5 seconds vs DreamBooth's ~10-minute training).
• Custom Diffusion (Kumari et al. 2023): only update cross-attention's $W_K, W_V$ (leaving $W_Q$ alone), with regularization images to prevent overfitting. Essentially a narrower LoRA-DreamBooth.

§8 Image Editing: SDEdit / InstructPix2Pix / Prompt-to-Prompt

8.1　SDEdit (Meng et al. 2022 ICLR)

Idea: image editing = "add partial noise to the input image → reverse back guided by the prompt".

   input image x (e.g. sketch)
         │
         │   noise_strength = 0.6 (example)
         ▼
   z_0 = VAE_enc(x)
         │
   z_τ = √(ᾱ_τ) z_0 + √(1 - ᾱ_τ) ε,   τ = noise_strength × T
         │
         ▼
   reverse SDE / ODE from t=τ to t=0    (guided by prompt c)
         │
         ▼
   z_0' →  VAE_dec → edited image x'

Key parameter strength $\in [0, 1]$:

• $\text{strength} \to 0$: very little noise; output $\approx$ input (no editing)
• $\text{strength} \to 1$: fully noised; output = pure text-to-image (input signal lost)
• Common range 0.3–0.8 for the right balance

8.2　SDEdit core code

import torch

@torch.no_grad()
def sdedit_sample(unet, vae, text_encoder, tokenizer, scheduler,
                  init_image, prompt, neg_prompt="",
                  strength=0.7, num_steps=30, cfg_scale=7.0,
                  device="cuda", dtype=torch.float16):
    assert 0.0 < strength <= 1.0, "strength=0 == no editing (return input directly); strength>1 is invalid"

    # 1) text encoding (same as §5)
    ids_pos = tokenizer(prompt, padding="max_length", max_length=77,
                        truncation=True, return_tensors="pt").input_ids.to(device)
    ids_neg = tokenizer(neg_prompt, padding="max_length", max_length=77,
                        truncation=True, return_tensors="pt").input_ids.to(device)
    emb = torch.cat([text_encoder(ids_neg)[0],
                     text_encoder(ids_pos)[0]], dim=0)

    # 2) encode the input image to latent
    z0 = vae.encode(init_image).latent_dist.sample() * 0.18215   # [1, 4, h, w]

    # 3) set timestep subset; only traverse the last `strength` portion
    scheduler.set_timesteps(num_steps, device=device)
    import math
    # use ceil + max(1, .) to ensure at least 1 step runs when strength>0
    n_edit = max(1, math.ceil(num_steps * strength))
    t_start = num_steps - n_edit               # 0 ≤ t_start < num_steps
    timesteps = scheduler.timesteps[t_start:]  # at least 1 timestep

    # 4) add noise to z_0 at the timesteps[0] level
    eps = torch.randn_like(z0)
    z = scheduler.add_noise(z0, eps, timesteps[:1])

    # 5) main loop (identical to §5)
    for t in timesteps:
        z_in = torch.cat([z, z], dim=0)
        z_in = scheduler.scale_model_input(z_in, t)
        eps_pred = unet(z_in, t, encoder_hidden_states=emb).sample
        eps_neg, eps_pos = eps_pred.chunk(2, dim=0)
        eps_cfg = eps_neg + cfg_scale * (eps_pos - eps_neg)
        z = scheduler.step(eps_cfg, t, z).prev_sample

    # 6) decode
    z = z / 0.18215
    return ((vae.decode(z).sample.clamp(-1, 1) + 1) / 2)

8.3　InstructPix2Pix (Brooks et al. 2023 CVPR)

Instruction-style editing: "add a hat to the dog". Training data is synthesized via GPT-3 + Prompt-to-Prompt (pairwise (source, instruction, target) tuples), used to fine-tune SD 1.x. Architecture:

• U-Net input channels 4 → 8 (original latent 4 + source image latent 4)
• Two CFG scales: text guidance $s_T$ and image guidance $s_I$, tuned independently

$$\hat\epsilon = \hat\epsilon(\emptyset, \emptyset) + s_I [\hat\epsilon(c_I, \emptyset) - \hat\epsilon(\emptyset, \emptyset)] + s_T [\hat\epsilon(c_I, c_T) - \hat\epsilon(c_I, \emptyset)]$$

8.4　Prompt-to-Prompt (Hertz et al. 2023 ICLR)

Training-free editing: achieves "change words but keep structure" by manipulating cross-attention maps.

• Run the original prompt P and record the per-layer per-step cross-attn maps $M_t$ (shape $[H_q, L_t]$ — weight each image token assigns to each text token)
• Run new prompt P* (changing only one word, e.g., cat → dog) but force-replace the attention map for the corresponding text token positions with the original P's
• Thus structure (which image patches attend to which text positions) is preserved; only content (the values aggregated after softmax) changes

Suitable for "swap one word", "add adjective", and "reweight emphasis" types of edits.

§9 Distillation: Few-Step Sampling

9.1　LCM / LCM-LoRA (Luo et al. 2023)

Latent Consistency Model: distill latent diffusion into a Consistency Model (Song et al. 2023), letting a single-step prediction $f_\theta(z_t, t)$ directly yield $z_0$ — compressing 50 steps to 4-8.

LCM-LoRA: package this distillation as a LoRA. Base SDXL/SDXL-1.0 + a ~200MB LCM-LoRA = 4-step generation. Zero-shot plug-in, stacks with personalization LoRA.

9.2　SDXL-Turbo: ADD (Sauer et al. 2024 ECCV) / SD3-Turbo: LADD (Sauer et al. 2024)

SDXL-Turbo = ADD (Adversarial Diffusion Distillation, arXiv 2311.17042 / ECCV 2024); SD3-Turbo = LADD (Latent Adversarial Diffusion Distillation, arXiv 2403.12015) — two distinct methods. ADD uses a pixel-space vision encoder as discriminator; LADD moves to a latent-space discriminator that scales to the SD3 model.

Adversarial Diffusion Distillation (ADD):

• Teacher: original SDXL (multi-step diffusion model)
• Student: same architecture as teacher, targeting 1-4 step generation

• Three losses:

  1. Distillation loss: MSE / LPIPS between student output and teacher's multi-step output
  2. Adversarial loss: DINOv2-based discriminator judges student outputs
  3. Score loss: SDS-style score distillation

ADD is one of the SOTA few-step distillation routes; FLUX schnell is based on similar ideas ("timestep distillation" + adversarial).

9.3　DMD / DMD2 / Hyper-SD short table

§10 Evaluation: FID / CLIP-Score / ImageReward / HPSv2 / PickScore

§11 Complexity / Resources

11.1　Training side (order-of-magnitude estimates)

11.2　Inference side (rough, implementation-dependent)

SD 1.5 512×512 (native resolution, latent 64×64×4):

• Per-step U-Net FLOPs: ~0.5T
• Per-step cross-attn QKV: (L_z=4096) × (L_t=77) → 0.3M score matrix × multi-layer
• Per-step memory: ~2-3 GB (FP16, no cross-attn KV cache)
• 30 steps ≈ 1.5-2 s / image (A100, FP16)

(SD 1.5 at 1024² is ~4× slower than 512²; the community does not recommend non-native-resolution direct inference, typically pairing it with SDXL or super-resolution.)

SDXL 1024×1024:

• Per-step U-Net FLOPs: ~1.2T
• 30 steps ≈ 4-5 s / image (A100, FP16)

FLUX.1-dev 1024×1024:

• MM-DiT FLOPs ≈ 2.5T / step
• 28 steps ≈ 12-15 s (A100); H100 ~5-7 s

11.3　Memory footprint cheat sheet

§12 Comparisons with Related Methods

12.1　Diffusion family vs other generative models

12.2　Internal routes within diffusion

§13 25 Frequently-Asked Interview Questions (codex 5.5 xhigh top-lab interviewer perspective)

Expand each entry to see key answer points + common pitfalls.

L1 must-know (asked at every vision / multimodal engineering interview)

L2 advanced (research / production positions)

L3 top-lab / deep questions (research-lead perspective)

image latent Q  ─►───┐
                     │   softmax(QK^T/√d) V
                     │
text K/V (frozen)  ──┘──► attended_out → added to image latent residual

                 [txt | img] concatenated into one sequence
                       │
        ┌──────────────┼──────────────┐
        │              │              │
  independent QKV (txt)  independent QKV (img)
        │              │              │
        └──────────────┼──────────────┘
                       ▼
              joint self-attention
                  (global attn matrix)
                       │
        ┌──────────────┼──────────────┐
        │              │              │
   text output updated    image output updated

§A Appendix: References

Main papers (chronological)

• DDPM — Ho, Jain, Abbeel 2020 NeurIPS
• LDM / Stable Diffusion — Rombach, Blattmann et al. 2022 CVPR
• Classifier-Free Guidance — Ho & Salimans 2022 (workshop / arXiv)
• DiT — Peebles & Xie 2023 ICCV
• U-ViT — Bao, Nie et al. 2023 CVPR
• LoRA — Hu, Shen et al. 2022 ICLR
• DreamBooth — Ruiz, Li et al. 2023 CVPR
• Textual Inversion — Gal, Alaluf et al. 2023 ICLR
• Custom Diffusion — Kumari, Zhang et al. 2023 CVPR
• HyperDreamBooth — Ruiz, Li et al. 2024 CVPR
• ControlNet — Zhang, Rao, Agrawala 2023 ICCV
• T2I-Adapter — Mou, Wang et al. 2024 AAAI
• IP-Adapter — Ye, Zhang et al. 2023 (arXiv 2308.06721)
• InstantID — Wang, Bai et al. 2024 (arXiv 2401.07519)
• PuLID — Guo, Wu et al. 2024 NeurIPS
• PhotoMaker — Li, Cao et al. 2024 CVPR
• SDEdit — Meng, He et al. 2022 ICLR
• InstructPix2Pix — Brooks, Holynski, Efros 2023 CVPR
• Prompt-to-Prompt — Hertz, Mokady et al. 2023 ICLR
• SDXL — Podell, English et al. 2024 ICLR
• SD3 — Esser, Kulal et al. 2024 ICML
• FLUX.1 — Black Forest Labs 2024 (technical report)
• PixArt-α / Σ — Chen, Yu et al. 2024 ICLR / ECCV
• Hunyuan-DiT — Zhimin Li, Jianwei Zhang et al. 2024 (arXiv 2405.08748)
• Imagen — Saharia, Chan et al. 2022 NeurIPS
• ADD / SDXL-Turbo — Sauer, Lorenz et al. 2024 ECCV (arXiv 2311.17042)
• LCM — Luo, Tan et al. 2023 (arXiv 2310.04378)
• LCM-LoRA — Luo, Tan et al. 2023 (arXiv 2311.05556)
• DMD — Yin, Gharbi et al. 2024 CVPR
• ImageReward — Xu, Liu et al. 2023 NeurIPS
• HPSv2 — Wu, Hao et al. 2023 (arXiv 2306.09341)
• PickScore / Pick-a-Pic — Kirstain, Polyak et al. 2023 NeurIPS
• FID — Heusel, Ramsauer et al. 2017 NeurIPS

One-sentence summary

This cheat sheet covers latent diffusion mathematics (VAE + DDPM/RF + CFG) through mainstream architecture evolution (SD 1.x → SDXL → SD3 → FLUX), conditioning systems (ControlNet / T2I-Adapter / IP-Adapter / InstantID), personalization (DreamBooth / Textual Inversion / LoRA / Custom Diffusion), editing (SDEdit / InstructPix2Pix / Prompt-to-Prompt), distillation (LCM / ADD / DMD), and evaluation (FID / CLIP-Score / ImageReward / HPSv2). 25 questions are split L1/L2/L3; L3 emphasizes the production-lab viewpoint (zero-conv chain-rule derivation, size conditioning training effect, MM-DiT information flow, LoRA Q/K/V choice, SDXL + LCM-LoRA + ControlNet + IP-Adapter co-deployment trade-offs).