---
category: 06-stable-diffusion-flux
category_title: "Stable Diffusion & Flux: Open-Weights Prompting, Conditioning, Training, and Local Pipelines"
angles:
  - { slug: 6a, title: "SD 1.5 / 2.1 / XL architecture and canonical prompting conventions" }
  - { slug: 6b, title: "Flux family prompting: guidance distillation, T5 semantics, Kontext editing, Flux.2" }
  - { slug: 6c, title: "ControlNet, IP-Adapter, T2I-Adapter, InstantID, PuLID, Reference-Only, Flux Tools" }
  - { slug: 6d, title: "LoRA / LoKr / LoHA / DoRA training for brand style, icon packs, asset consistency" }
  - { slug: 6e, title: "Local inference pipelines: Diffusers, ComfyUI, Fooocus, InvokeAI, A1111/Forge, SwarmUI" }
primary_source_count: 80
status: synthesis
last_updated: 2026-04-19
tags: [stable-diffusion, sdxl, flux, flux2, controlnet, ip-adapter, lora, dora, comfyui, diffusers, asset-generation, brand-consistency]
---

> **📅 Research snapshot as of 2026-04-19.** Provider pricing, free-tier availability, and model capabilities drift every quarter. The router reads `data/routing-table.json` and `data/model-registry.json` at runtime — treat those as source of truth. If this document disagrees with the registry, the registry wins.

# 06 — Stable Diffusion & Flux: Category Index

## Category Executive Summary

Across five angles (6a SD 1.5/2.1/XL, 6b Flux family, 6c conditioning adapters, 6d LoRA training, 6e local pipelines), open-weights image generation in 2024–2026 has bifurcated into two architecturally different stacks with very different prompting contracts, and a prompt-to-asset targeting software assets must handle both. The fifteen load-bearing insights:

1. **Stable Diffusion prompting is cross-attention manipulation, not English parsing.** Every A1111 DSL feature — `(word:1.2)`, `[from:to:when]`, `AND`, `BREAK` — is a direct operation on the 77-token CLIP tensor fed into UNet cross-attention [6a]. Porting prompts between UIs (A1111, ComfyUI, InvokeAI, Forge) fails because of weight-normalization differences, not syntax differences; `BlenderNeko/ComfyUI_ADV_CLIP_emb` with `token_normalization=mean` is the portability fix [6a].
2. **Flux rejects `negative_prompt` by design, not by oversight.** Flux.1 [dev/schnell/pro], Flux.1 Tools, Kontext, and Flux.2 are all guidance-distilled — CFG is baked into the student weights as a conditioning scalar, so there is no unconditional branch for a negative prompt to hijack [6b]. BFL's Flux.2 prompting guide makes this explicit; any enhancer that silently passes a `negative_prompt` to a Flux endpoint will either 400 or drop it [6b].
3. **Long, prose prompts are how Flux actually outperforms SDXL.** T5-XXL carries 512 tokens on Flux.1 and Mistral-3 24B VLM carries ~32K-effective on Flux.2 vs. CLIP's 77; BFL's Subject+Action+Style+Context scaffold plus JSON prompts with hex-coded subjects (`"Apple: #0047AB"`) are the structural replacements for tag-soup + negative-prompt hygiene [6b].
4. **SD wins when flat-vector / duotone / illustrated aesthetics matter and VRAM is tight; Flux wins when text rendering, identity preservation, and prompt adherence matter.** Flux.1 [pro] sits second on the Artificial Analysis typography benchmark (~1068 ELO, behind Ideogram 3) and renders correct text where SDXL reliably hallucinates characters [6b]. But Flux tilts photographic, so flat-vector icon LoRAs are often easier to land on SDXL [6d].
5. **SDXL's dual encoder is a tool, not a redundancy.** Splitting `text_g` (OpenCLIP-bigG, 1280-dim) for composition/mood and `text_l` (CLIP-L, 768-dim) for colors, materials, named entities yields measurably better SDXL output than sending the same prompt twice [6a]. A1111 abstracts this away; ComfyUI's `CLIPTextEncodeSDXL` and micro-conditioning inputs (`original_size`, `crops_coords_top_left`, `target_size`) expose it [6a].
6. **`BREAK` is the single most under-used tool for asset prompting.** Because each 75-token CLIP chunk is encoded in isolation, putting subject tokens before `BREAK` and background tokens after prevents "indigo-to-violet gradient" from leaking onto the background you asked to keep white — the exact bug that wrecks logo/icon jobs on SDXL [6a]. Civitai's top logo prompts all use `BREAK`.
7. **Guidance-distilled models pay for faster inference with fragile artifact recipes.** The `"blurry, low-quality, watermark"` SDXL ritual has no direct translation to Flux. The working substitutes are (a) affirmative rewrites ("tack-sharp, clean un-watermarked composition, anatomically correct hands"), (b) `prompt_upsampling=true` on Flux.2 to pre-enrich terse inputs, and (c) routing edits through Kontext instead of re-rolling [6b].
8. **ControlNet + IP-Adapter is the canonical two-adapter stack for brand-consistent asset families.** Canny (weight 0.6) locks silhouette while IP-Adapter-Plus (weight 0.4) carries color/material language — the single most reliable open-source recipe for "16 sibling icons in one brand system from one logo" across SDXL, Flux.1, and Flux.2 [6c]. Stacking more than two adapters usually degrades quality.
9. **First-party Flux Tools dominate community adapters where they overlap.** Flux.1 Fill / Canny / Depth / Redux (Nov 2024) ship as BFL-trained guidance-distilled checkpoints and outperform community InstantX / XLabs adapters on their native modes; use InstantX/Shakker ControlNet-Union only for modes BFL doesn't ship (pose) [6c]. Kontext further collapses the edit loop — "logo → favicon → OG card → splash" becomes one conversation, not a re-roll-and-mask cycle [6c].
10. **PuLID is strictly better than InstantID for human identity on SDXL; PuLID-Flux is the Flux default.** PuLID's contrastive alignment keeps non-ID attributes (pose, style, background) close to the base model's un-identity-conditioned output, so style degradation is dramatically lower than InstantID [6c]. For non-human mascots, IP-Adapter-Plus at 0.7–0.9 + optional ControlNet-OpenPose is the equivalent recipe [6c].
11. **LoRA is the lowest-risk, highest-leverage brand-consistency primitive.** 10–30 curated images, 1–4 hours on a 24 GB GPU (or $2–6 on Replicate/fal), rank 8 for flat icons / rank 16 with DoRA for painterly / rank 4 LoHA for logo marks — ships as a 20–400 MB adapter that slots into any SDXL or Flux pipeline [6d]. Flux + ostris `ai-toolkit` at rank 16–32 on 15–25 captioned images with a rare trigger token is the 2026 default.
12. **DoRA is a free quality upgrade over vanilla LoRA for style work.** Weight-Decomposed Low-Rank Adaptation (Liu et al. 2024) splits weights into magnitude and direction and applies the low-rank update only to the direction; at the same rank it gives sharper color fidelity and less style-leak into unrelated concepts, at ~2× training cost and zero inference cost [6d]. LoKr is the pick for <10 MB edge-shippable artifacts; LoHA helps on 5–10-image micro-datasets.
13. **Dataset and caption discipline dominate hyperparameter choice.** Rank 16 / alpha 16 is a safe LoRA default regardless of target; pushing rank to 64 won't rescue a noisy 200-image set or captions that leak the style into prose [6d]. The rule is: caption the *content*, let the rare trigger token own the style, sprinkle 5–10% caption dropout.
14. **ComfyUI behind `POST /prompt` is the production default for a multi-asset pipeline; Diffusers is the default when you need single-box throughput.** ComfyUI's typed node graph gives day-of access to every new adapter (Flux Tools, Kontext, LayerDiffuse, IP-Adapter-Plus) without code changes; Diffusers with `torch.compile` + `enable_model_cpu_offload` + bnb-4bit gives the tightest in-process control for 64-image batches; SwarmUI is the only open-source way to fan a single request across N backends cleanly [6e].
15. **Distilled-model pitfalls are real and concentrate in three places.** (a) Flux.1 [schnell]'s ADD step-distillation to 1–4 steps means `guidance_scale` is inert; (b) Flux.1 [dev]'s guidance distillation means no negatives and values above ~6 saturate / break hands; (c) stacking two strong adapters on distilled models forces weight reductions (−0.1 to −0.2 vs. SDXL equivalents) because the single forward pass has less headroom to absorb conflicting signals [6b][6c].

## Map of the Angles

- **[6a] SD 1.5 / 2.1 / SDXL architecture and canonical prompting conventions** — Latent-diffusion architecture primer, full A1111 prompt-DSL reference (weights, scheduling, alternation, composable-diffusion `AND`, `BREAK` chunk boundaries), SDXL dual-encoder split (`text_g` vs `text_l`) and micro-conditioning (`original_size`, `crops_coords_top_left`, `target_size`, refiner `aesthetic_score`), `clip_skip` per-model rules, Civitai practitioner patterns for icon/logo generation.
- **[6b] Flux family prompting: guidance distillation, T5 semantics, Kontext editing, Flux.2** — Complete BFL model-family table (Flux.1 [pro/dev/schnell], 1.1 [pro]/Ultra, Tools, Finetuning API, Kontext [pro/max/dev], Flux.2 [pro/flex/dev 32B/klein] with Mistral-3 24B VLM). Explains guidance distillation as conditioning-input (not extrapolation coefficient), documents the "no negative prompt" constraint with affirmative-rewrite tables, text-rendering benchmarks, Kontext edit-loop primitives, Flux.2 JSON-prompt + hex-color + multi-reference affordances, fal/Replicate/Runware schema parity.
- **[6c] ControlNet, IP-Adapter, T2I-Adapter, InstantID, PuLID, Reference-Only, Flux Tools** — Primers on each adapter family (Zhang ControlNet zero-conv injection, Ye IP-Adapter decoupled cross-attention, InstantID IdentityNet+FaceID, PuLID contrastive alignment + Lightning T2I), seven concrete asset-consistency recipes (silhouette-locked icon family, logo-stamping, shape-locked variants, mascot/character consistency, brand palette transfer, room retexture, Reference-Only fallback), weight-stacking rules, failure modes including identity-transfer regulation.
- **[6d] LoRA / LoKr / LoHA / DoRA training for brand style, icon packs, asset consistency** — PEFT math (LoRA rank-r decomposition, DoRA magnitude/direction split, LoKr Kronecker for 10–30× smaller files, LoHA Hadamard for micro-datasets). Toolchain comparison (ostris `ai-toolkit` as Flux default, kohya `sd-scripts` as SDXL workhorse, PEFT, Replicate + fal.ai as $2–6 cloud trainers). Four dataset recipes with per-recipe rank/alpha/steps/LR/augmentation. Captioning strategy, style-lock patterns at inference, block-weight merging, compute/cost table.
- **[6e] Local inference pipelines for asset batches** — Six-runtime comparison matrix (Diffusers, ComfyUI, SwarmUI, InvokeAI, A1111, Forge, Fooocus). Full API surface for each (ComfyUI `/prompt`+WS API-format JSON, InvokeAI `enqueue_batch` zipped/combinatorial, SwarmUI `GenerateText2ImageWS` fan-out, A1111/Forge `/sdapi/v1/txt2img` + `override_settings`, Forge NF4/GGUF, Fooocus-API). Four batch recipes. VRAM profile table for Flux across FP16/FP8/GGUF Q8/Q5/Q4/NF4. RunPod serverless `worker-comfyui` + Apatero template.

## Cross-Cutting Patterns

- **Two architectures, one enhancer contract.** SD (UNet + CLIP) and Flux (MM-DiT + T5/VLM) require different surface prompts but the enhancer's job is the same: produce subject/background separation, lock palette, normalize negatives, pick adapters. 6a, 6b, and 6d all converge on the idea that the enhancer should branch on `{family: "sd" | "flux"}` once and then apply model-family-appropriate rewrites downstream.
- **Affirmative descriptions beat prohibitions on both families.** On Flux this is forced (no negative prompt); on SDXL this is still better practice because negative prompts consume cross-attention bandwidth and A1111's renormalization reallocates it away from positives. 6a and 6b agree: "clean un-watermarked composition" beats `"watermark, text, signature"`.
- **Subject/background separation via explicit tokens.** SDXL uses `BREAK` [6a]; Flux uses prose sentence boundaries ("on a pure white background, centered with strong negative space") [6b]; both rely on the same mechanism — forcing the text encoder to segregate subject tokens from background tokens so that cross-attention spatial regions attend cleanly.
- **Silhouette + style is the durable two-channel composition.** Canny ControlNet (shape) + IP-Adapter-Plus (style) on SDXL [6c]; Flux Canny + Redux on Flux [6c]; ControlNet + LoRA trigger token [6c+6d]. The pattern generalizes: one conditioning signal for geometry, one for appearance.
- **1024×1024 is the modern native.** SDXL multi-aspect bucketing is weakest at sub-native [6a]; Flux.1 [dev] and Flux.2 both target 1024² or higher natively [6b]; LoRA training uniformly uses 1024² [6d]; ComfyUI/Diffusers batch recipes default to 1024² [6e]. Anything below loses detail to bucketing and VAE decode artifacts.
- **No model natively outputs reliable RGBA.** Every angle hands off transparency to a matting post-process: SDXL via `LayerDiffuse`, Flux via RMBG-1.4 / BiRefNet, and ComfyUI's LayerDiffuse node is the cleanest integrated path [6b][6d][6e]. See category 13 for the deep-dive.
- **LoRA + adapter stacking requires weight coordination.** `<lora:brand:0.9>` + IP-Adapter 0.5 on SDXL works; the Flux equivalent needs both weights reduced ~0.1–0.2 because of the distilled single forward pass having less headroom [6c][6d].

## Controversies / Disagreements

- **"Flux can do negative prompts with a workaround" vs. "No, it really can't."** BFL's Flux.2 prompting guide [6b ref 10] is unambiguous: no negative prompts. But community work — Forge's "Flux1D Distillation CFG" experiment and semi-de-distillation via inverse-cosine timestep remapping [6b ref 18] — shows negatives can be partially re-enabled by forcing a second forward pass and perturbing the schedule. 6b treats these as research curiosities, not production features, because no commercial endpoint exposes them and they cost 2× inference time. 6d notes in passing that "Flux ignores CFG-style negatives … though flux-dev does honor a weak negative via guidance_scale tricks," which is the softer framing. The consensus is: treat negatives as not-working for product code, surface the affirmative-rewrite table to users, and leave the hack to power users.
- **ComfyUI-Union ControlNet vs. specialized per-mode ControlNets.** 6c explicitly flags that "Union models are weaker per-mode" and that Jasperai's single-mode Flux Canny often beats InstantX Union on canny. But Union is cheaper to load/swap in pipelines. The recommendation splits: use Union for dev iteration, swap in a specialized for production quality if the pipeline is fixed to one mode.
- **LoRA rank for style: "start at 16" vs. "start at 8 for icons."** 6d recommends rank 16 as the universal safe default but rank 8 specifically for flat-vector icon packs (Recipe A), because higher rank overfits and memorizes specific icons rather than the style. This is the one place where the "just use 16" heuristic is wrong.
- **A1111 vs. ComfyUI weight semantics.** 6a documents that A1111 normalizes token weights across the prompt while ComfyUI does not, so identical prompts + seeds produce different images [6a ref 9]. The consensus fix is `ComfyUI_ADV_CLIP_emb` with `token_normalization=mean`, but this is not the ComfyUI default, so any cross-UI prompt-library must carry normalization metadata, not just the prompt string.
- **"Just add more images" vs. dataset quality for brand LoRAs.** 6d's headline heuristic — "a curated 15-image set beats a 200-image noisy set at any rank" — pushes back against Civitai community practice. Disagreement is about regularization: SDXL benefits from 50–200 regularization images; Flux community says skip them. Both are true, but only if dataset curation is actually done.
- **SD 1.5 is dead vs. SD 1.5 is still the right tool for flat icons.** 6a and 6d both note SD 1.5's continued utility for anime/illustration/flat-vector checkpoints where SDXL over-indexes on photoreal and Flux tilts further. No modern SOTA benchmark reports SD 1.5 favorably, but the community LoRA ecosystem is richest there.

## Gaps

- **No dedicated treatment of inpainting/outpainting prompt patterns within this category.** 6c covers Flux.1 Fill at the adapter level but prompt-side masking ergonomics (mask-aware prompt design, region-specific prompts via Regional Prompter or ComfyUI Impact-Pack) is not deep-dived. Cross-reference category 16 and category 18.
- **Aspect-ratio and multi-scale guidance for OG/Twitter/favicon targets.** The angles mention 1024² but platform-specific ratios (1200×630 OG, 1080×1080 Instagram, 180×180 Apple touch icon) are handled in category 11 / 18. The enhancer should own the mapping; this category doesn't.
- **VAE choice and color-accuracy impact on brand palette fidelity.** 6a documents VAE as "the pixel-space autoencoder" but doesn't cover the SDXL VAE 0.9 vs. 1.0 vs. madebyollin SDXL-FP16 VAE choice, which materially affects color accuracy for brand work.
- **Serving economics.** 6e gives VRAM profile and cloud options but not a full cost-per-1000-assets comparison across Replicate vs. fal vs. self-hosted A100 vs. RunPod spot. The plugin roadmap needs this.
- **Quantitative benchmarks for text rendering inside logos/wordmarks.** 6b cites ELO and correctness ratios but doesn't include a reproducible test harness the plugin could use as CI.
- **Flux.2 ecosystem coverage now improving.** Flux.2 launched 2025-11-25 and ComfyUI support landed at launch via NVIDIA-collaborated FP8 path; 6e's matrix still assumes Flux.1 for most rows but has been updated with coverage notes.
- **Flux.2 [klein] (released Jan 2026) unifies T2I and editing** in a single compact architecture — 4B (Apache 2.0, ~13 GB VRAM) and 9B (Non-Commercial, ~29 GB VRAM) — partially replacing the need for separate Kontext and Fill tools on Flux.2. Klein is not yet covered in depth in the angle files.
- **SD 3.5 prompting contract not covered.** 6a covers SD 1.5/2.1/XL; SD 3.5 (released Oct 2024, 8B Large + 2.6B Medium) has a different three-encoder architecture (CLIP-L + OpenCLIP-bigG + T5-XXL) closer to Flux's prose style than SDXL tag-soup. No angle covers SD 3.5-specific prompting patterns in depth.
- **Agentic use of Kontext as an iterative edit tool.** 6b flags Kontext as "the primitive that makes iterative asset generation economically viable" but there's no worked example of a multi-turn agentic loop (generate → critique → Kontext-edit → accept). This connects to category 19.
- **Legal/licensing coverage.** FLUX-1 Non-Commercial, FLUX-2 Non-Commercial, Apache 2.0 (schnell, Flux.2-klein-4B, VAE) matter for the plugin's commercial posture; 6b lists licenses but doesn't translate them into product decisions. Note: Flux.2 [klein] 4B is Apache 2.0 (commercial-safe); the 9B variant is Non-Commercial.

> **Updated 2026-04-21:** Several gaps above are now partially closed. Flux.2 [klein] 4B (Apache 2.0) is now the recommended local consumer-GPU path for Flux.2-quality output, not a "forthcoming" model. AUTOMATIC1111 is maintenance-only (last release Jul 2024) — the "A1111 as a viable option" framing in the self-hostable pipeline recommendation should be understood as "legacy" for users with existing SD 1.5/SDXL workflows only. Fooocus is LTS/SDXL-only. kohya sd-scripts Flux training is no longer experimental — it's in main via `flux_train_network.py`.

## Actionable Recommendations for the Plugin

### Self-hostable pipeline recommendation

- **Default runtime: ComfyUI behind `POST /prompt`** [6e], running on a 24 GB GPU (4090 / A5000 / L40S) or the `runpod/worker-comfyui` serverless image for bursty workloads. Ship a versioned library of API-format workflow JSON templates per asset category (logo, app icon, illustration, OG card, favicon, splash). Parametrize by (a) prompt text node, (b) seed, (c) filename_prefix, (d) checkpoint path, (e) LoRA stack, (f) adapter weights. Install the must-have custom-node set documented in 6e: `ComfyUI-Manager`, `rgthree-comfy`, `ComfyUI-Impact-Pack`, `was-node-suite`, `ComfyUI-GGUF`, `ComfyUI-IPAdapter-plus`, `ComfyUI-ControlNet-Aux`, `ComfyUI-LayerDiffuse`.
- **Low-VRAM path: Forge with NF4 Flux + A1111-shape API** [6e]. Any existing `sdwebuiapi` client works unchanged. Useful for enthusiast tier / user-owned GPU (12–16 GB).
- **High-throughput path: Diffusers in-process with `torch.compile` + `enable_model_cpu_offload` + bnb-4bit** [6e]. 64-image batches via `itertools.product` over (subject × seed × style × aspect) with inner `BATCH=4`. Wrap in `asyncio.Queue` for request-level concurrency.
- **Multi-box path: SwarmUI fronting N ComfyUI backends** [6e]. One session per plugin instance; `GenerateText2ImageWS` fans the request across backends automatically.
- **Model defaults**: Flux.1 [dev] for typography / identity / prompt-adherence jobs; Flux.1 [schnell] for fast iteration; SDXL (Juggernaut XL / RealVis XL for photoreal, Vector illustration SDXL LoRA stack for flat assets) for flat/vector icon work and where VRAM is tight; Flux.2 [pro]/[flex] via API for maximum quality / multi-reference; **Flux.2 [klein] 4B** (Apache 2.0, ~13 GB VRAM, sub-second inference) as the new local consumer-GPU default when Flux.2-quality output is needed.

> **Updated 2026-04-21:** **Flux.2 [klein]** shipped January 15, 2026 (not "beta forthcoming") and is the practical local choice for Flux.2-quality output on consumer hardware. The "wait for Flux.2 ecosystem to stabilize" caveat from the original synthesis is now resolved — ComfyUI, diffusers, and InvokeAI all have production-ready Flux.2 support. **AUTOMATIC1111** is maintenance-only (last release v1.10.1, July 2024); remove it from active recommendations — point users to Forge instead. **Fooocus** is SDXL-only LTS; not suitable for Flux or SD3.5 workflows.

### ControlNet / IP-Adapter strategy

- **Expose an "Asset consistency mode"** [6c] that toggles between four pre-wired adapter stacks: (1) **Silhouette-locked icon family**: Canny (0.6) + IP-Adapter-Plus (0.4) from a single reference logo. (2) **Logo stamping**: scene without logo → Flux Fill with rectangular mask + XLabs Flux IP-Adapter v2 (0.9) pointing at the logo asset. (3) **Shape-locked color variants**: Flux Canny (0.8) from binary silhouette mask + prompt-varied material/color. (4) **Brand palette transfer**: Flux Redux (0.7–1.0) from a 4–6-image style sheet, optionally stacked with XLabs Flux IP-Adapter v2 at 0.3.
- **Cap stacks at two adapters** [6c]. Surface a UI warning if a user tries to add a third; internally enforce weight re-normalization to keep sum ≤ 1.3.
- **Prefer first-party Flux Tools** [6c]: BFL Canny / Depth / Fill / Redux over community InstantX/XLabs for their native modes. Fall back to InstantX Union / Shakker Union only for modes BFL doesn't ship (pose).
- **Never use the base model to render logo text** [6c]. Generate the scene, composite the vector logo via image-editor post-process, or use Flux Fill with IP-Adapter pointed at the clean logo PNG.
- **Human identity: PuLID-Flux by default** [6c]; InstantID as SDXL fallback. Hard-disable identity adapters unless the user has uploaded a consented reference (regulatory posture for EU AI Act + US state laws).

### LoRA training hook

- **Ship a "Train a brand LoRA" flow** [6d] that accepts 10–30 images and runs one of three presets: (A) flat-vector icon pack (rank 8, alpha 8, 2k steps, Flux ostris `ai-toolkit` or SDXL kohya), (B) painterly marketing illustration (rank 16 with DoRA, 4k steps), (C) duotone brand (rank 8–16 with hex-anchored captions), (D) logo mark (LoHA rank 4 with 50–100 regularization images).
- **Backend**: Replicate `ostris/flux-dev-lora-trainer` or fal.ai `flux-lora-fast-training` for $2–6 per brand, with an optional RunPod-H100 + ai-toolkit queue path for self-hosted. For SDXL LoRAs, kohya `sd-scripts` (main branch, `flux_train_network.py` for Flux — no longer experimental) is the alternative; kohya GUI (bmaltais) exposes the full parameter surface.
- **Captioning pipeline**: auto-caption with BLIP-2 / CogVLM → strip style words → inject rare trigger token as first word → expose the captions for user review before training kicks off. This is the single biggest quality lever [6d].
- **Inference integration**: the enhancer must know (a) the trigger token for each brand LoRA, (b) the LoRA weight range (default 0.8–1.0 Flux, 0.65–0.9 SDXL), (c) the asset-template whitelist (flat icon vs. illustration) the LoRA was trained for. Store these as metadata alongside the safetensors file.
- **Multi-brand serving**: keep the base model shared; hot-swap adapters via `diffusers.set_adapters()` or ComfyUI Power Lora Loader (rgthree). Don't merge-and-unload per-brand unless the brand is so high-volume it warrants its own dedicated endpoint — loss of dynamic scale control outweighs the ~2% inference speedup.
- **DoRA by default for painterly styles; LoKr when artifact size matters for edge/CDN** [6d].

### Prompt-enhancer contract (family-aware)

- Detect `{family: "sd" | "flux"}` from the configured pipeline. For **SD**: emit A1111-DSL with `BREAK` subject/background separation, weighted quality-tag header for illustration checkpoints, `clip_skip=2` for NAI-lineage anime/flat, `(masterpiece:1.2), (best quality:1.3)` prefix only on checkpoints that re-learned it. Split SDXL prompts into `text_g` (composition) and `text_l` (attributes) when the backend supports it. For **Flux**: emit Subject+Action+Style+Context prose, convert negatives to affirmatives, default `guidance_scale=3.5`, set `max_sequence_length=512`; on Flux.2 emit JSON with `color_palette` hex codes and offer `prompt_upsampling=true` for short inputs.

## Primary Sources Aggregated

**Core papers.** Rombach et al. *High-Resolution Image Synthesis with Latent Diffusion Models* ([arXiv:2112.10752](https://arxiv.org/abs/2112.10752)); Podell et al. *SDXL* ([arXiv:2307.01952](https://arxiv.org/abs/2307.01952)); Zhang et al. *ControlNet* ([arXiv:2302.05543](https://arxiv.org/abs/2302.05543)); Mou et al. *T2I-Adapter* ([arXiv:2302.08453](https://arxiv.org/abs/2302.08453)); Ye et al. *IP-Adapter* ([arXiv:2308.06721](https://arxiv.org/abs/2308.06721)); Wang et al. *InstantID* ([arXiv:2401.07519](https://arxiv.org/abs/2401.07519)); Guo et al. *PuLID* ([arXiv:2404.16022](https://arxiv.org/abs/2404.16022)); Hu et al. *LoRA* ([arXiv:2106.09685](https://arxiv.org/abs/2106.09685)); Liu et al. *DoRA* ([arXiv:2402.09353](https://arxiv.org/abs/2402.09353)); Yeh et al. *LyCORIS / LoKr / LoHA* ([arXiv:2309.14859](https://arxiv.org/abs/2309.14859)).

**Official vendor documentation.** Black Forest Labs [Flux.1 announcement](https://blackforestlabs.ai/announcing-black-forest-labs/), [Flux.1.1 pro](https://blackforestlabs.ai/announcing-flux-1-1-pro-and-the-bfl-api/), [Flux.1 Tools](https://blackforestlabs.ai/flux-1-tools/), [Flux Kontext](https://blackforestlabs.ai/blog/flux-1-kontext), [Flux.2](https://bfl.ai/blog/flux-2), [Flux Pro Finetuning](https://blackforestlabs.ai/blog/25-01-16-finetuning), [Kontext prompting guide](https://docs.bfl.ai/guides/prompting_guide_kontext_i2i), [Flux.2 prompting guide](https://docs.bfl.ai/guides/prompting_guide_flux2); HuggingFace model cards for [FLUX.1-dev](https://huggingface.co/black-forest-labs/FLUX.1-dev), [FLUX.1-schnell](https://huggingface.co/black-forest-labs/FLUX.1-schnell), [FLUX.1-Redux-dev](https://huggingface.co/black-forest-labs/FLUX.1-Redux-dev), [SDXL refiner](https://huggingface.co/stabilityai/stable-diffusion-xl-refiner-1.0), [SD 2.1 base](https://huggingface.co/stabilityai/stable-diffusion-2-1-base); [BFL reference repo](https://github.com/black-forest-labs/flux) and [flux2 repo](https://github.com/black-forest-labs/flux2).

**HF Diffusers.** [Flux pipeline](https://huggingface.co/docs/diffusers/main/en/api/pipelines/flux), [SDXL pipeline](https://huggingface.co/docs/diffusers/main/api/pipelines/stable_diffusion/stable_diffusion_xl), [AutoPipeline](https://huggingface.co/docs/diffusers/en/api/pipelines/auto_pipeline), [PEFT LoRA methods](https://huggingface.co/docs/peft/task_guides/lora_based_methods), [multi-adapter inference](https://huggingface.co/docs/diffusers/tutorials/using_peft_for_inference); diffusers issues [#2136 (77-token chunking)](https://github.com/huggingface/diffusers/issues/2136), [#2201 (padding differences)](https://github.com/huggingface/diffusers/issues/2201), [#7970 (CPU offload leak)](https://github.com/huggingface/diffusers/issues/7970); PR [#4901 (CLIP-skip)](https://github.com/huggingface/diffusers/pull/4901).

**UIs & tooling.** [A1111 Features wiki](https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Features), [A1111 prompt_parser.py](https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/master/modules/prompt_parser.py), [A1111 API wiki](https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/API), [Forge](https://github.com/lllyasviel/stable-diffusion-webui-forge), [sdwebuiapi](https://github.com/mix1009/sdwebuiapi); [ComfyUI](https://github.com/comfyanonymous/ComfyUI), [ComfyUI comms routes](https://docs.comfy.org/development/comfyui-server/comms_routes), [CLIPTextEncodeSDXL docs](https://docs.comfy.org/built-in-nodes/ClipTextEncodeSdxl), [ComfyUI discussions #521](https://github.com/Comfy-Org/ComfyUI/discussions/521) and [#1047](https://github.com/Comfy-Org/ComfyUI/discussions/1047); [ComfyUI-Manager](https://github.com/ltdrdata/ComfyUI-Manager), [rgthree-comfy](https://github.com/rgthree/rgthree-comfy), [Impact-Pack](https://github.com/ltdrdata/ComfyUI-Impact-Pack), [WAS nodes](https://github.com/WASasquatch/was-node-suite-comfyui), [ComfyUI-GGUF](https://github.com/city96/ComfyUI-GGUF), [ADV_CLIP_emb](https://github.com/BlenderNeko/ComfyUI_ADV_CLIP_emb), [IPAdapter_plus](https://github.com/cubiq/ComfyUI_IPAdapter_plus), [PuLID_ComfyUI](https://github.com/cubiq/PuLID_ComfyUI), [x-flux-comfyui](https://github.com/XLabs-AI/x-flux-comfyui); [InvokeAI](https://github.com/invoke-ai/InvokeAI) + [workflow docs](https://invoke-ai.github.io/InvokeAI/contributing/frontend/workflows/) + [invokeai-py-client](https://github.com/CodeGandee/invokeai-py-client); [SwarmUI API docs](https://github.com/mcmonkeyprojects/SwarmUI/blob/master/docs/API.md); [Fooocus](https://github.com/lllyasviel/Fooocus) + [Fooocus-API](https://github.com/mrhan1993/Fooocus-API).

**Training.** [kohya-ss/sd-scripts](https://github.com/kohya-ss/sd-scripts), [bmaltais kohya_ss GUI](https://github.com/bmaltais/kohya_ss), [LyCORIS](https://github.com/KohakuBlueleaf/LyCORIS), [ostris/ai-toolkit](https://github.com/ostris/ai-toolkit) + [Flux notebook](https://github.com/ostris/ai-toolkit/blob/main/notebooks/FLUX_1_dev_LoRA_Training.ipynb); [Replicate flux-dev-lora-trainer](https://replicate.com/ostris/flux-dev-lora-trainer), [fal.ai flux-lora-fast-training](https://fal.ai/models/fal-ai/flux-lora-fast-training).

**Adapter checkpoints.** [InstantX/FLUX.1-dev-Controlnet-Union](https://huggingface.co/InstantX/FLUX.1-dev-Controlnet-Union), Shakker-Labs FLUX.1-dev-ControlNet-Union-Pro, [XLabs-AI/flux-ip-adapter-v2](https://huggingface.co/XLabs-AI/flux-ip-adapter-v2), [jasperai/Flux.1-dev-Controlnet-Canny](https://huggingface.co/jasperai/Flux.1-dev-Controlnet-Canny), `h94/IP-Adapter` + `h94/IP-Adapter-FaceID`, [guozinan/PuLID](https://github.com/ToTheBeginning/PuLID).

**Serving & benchmarks.** [PyTorch torch.compile + Diffusers](https://pytorch.org/blog/torch-compile-and-diffusers-a-hands-on-guide-to-peak-performance/); [RunPod worker-comfyui](https://github.com/runpod-workers/worker-comfyui) + [deployment docs](https://docs.runpod.io/tutorials/serverless/comfyui); [Apatero RunPod template](https://apatero.com/blog/comfyui-docker-setup-that-just-works-runpod-template-2025); [InsiderLLM Flux local guide](https://insiderllm.com/guides/flux-locally-complete-guide/); Replicate Flux endpoints ([1.1-pro-ultra](https://replicate.com/black-forest-labs/flux-1.1-pro-ultra/api), [flux-pro-finetuned](https://replicate.com/black-forest-labs/flux-pro-finetuned/readme), [flux-schnell](https://replicate.com/black-forest-labs/flux-schnell/readme)); fal.ai [Schnell](https://fal.ai/docs/model-api-reference/image-generation-api/flux-schnell) and [Dev](https://fal.ai/docs/model-api-reference/image-generation-api/flux-dev) APIs; [WaveSpeed text-in-image benchmark](https://wavespeed.ai/blog/posts/qwen-2512-vs-sdxl-flux-text-benchmark); [ndurner Flux typography tests](https://ndurner.github.io/flux-text).

**Community / corroboration.** Civitai [advanced prompting syntax](https://civitai.com/articles/15104/advanced-on-site-prompting-syntax-composable-diffusion-prompt-scheduling-and-prompt-alternation), [Flux semi-dedistillation](https://civitai.com/articles/9984), [Vector illustration SDXL LoRA](https://civitai.com/models/60132); [myByways SDXL dual-encoder](https://mybyways.com/blog/two-text-prompts-text-encoders-in-sdxl-1-0); [MyAIForce PuLID vs InstantID](https://myaiforce.com/flux-pulid-vs-ecomid-vs-instantid/); [InvokeAI PR #7070 Flux ControlNet](https://github.com/invoke-ai/InvokeAI/pull/7070).
