Complete Guide to Text-to-Image Model Training Methodologies: From GAN to Flow Matching

[Text-to-Image Model Evolution Timeline]

2014-2019          2017-2020          2020-2023              2023-Present
    |                  |                  |                      |
   GAN              VAE/VQ-VAE        Diffusion Models      Flow Matching
    |                  |                  |                  + DiT
    v                  v                  v                      v
 ┌──────────┐    ┌──────────┐    ┌────────────────┐    ┌──────────────┐
 │StackGAN  │    │ VQ-VAE   │    │ DDPM (2020)    │    │ SD3 (2024)   │
 │AttnGAN   │    │ VQ-VAE-2 │    │ DALL-E 2(2022) │    │ Flux (2024)  │
 │StyleGAN  │    │ dVAE     │    │ Imagen (2022)  │    │ Pixart-Sigma │
 │BigGAN    │    │          │    │ SD 1.x (2022)  │    │              │
 │GigaGAN   │    │          │    │ SDXL (2023)    │    │              │
 └──────────┘    └──────────┘    └────────────────┘    └──────────────┘

Features:                Features:              Features:                   Features:
- Adversarial       - Discrete         - Iterative             - Straight paths
  Training            Latent Space       Denoising             - ODE-based
- Mode Collapse     - Codebook         - Classifier-Free      - Fewer steps
  issues                Learning           Guidance              - DiT backbone
- Fast generation         - Two-stage        - Latent Space          - Scalable
                      Training         - U-Net backbone

min_G max_D  V(D, G) = E_{x~p_data}[log D(x)] + E_{z~p_z}[log(1 - D(G(z)))]

- G (Generator): 랜덤 노이즈 z로부터 이미지 생성
- D (Discriminator): 실제 이미지와 생성 이미지 구분
- 학습 목표: G는 D를 속이고, D는 정확히 판별

# StyleGAN2 core training loop (simplified)
for real_images, _ in dataloader:
    # 1. Discriminator training
    z = torch.randn(batch_size, latent_dim)
    fake_images = generator(z)

    d_real = discriminator(real_images)
    d_fake = discriminator(fake_images.detach())

    d_loss = F.softplus(-d_real).mean() + F.softplus(d_fake).mean()

    # R1 Regularization (lazy: every 16 steps)
    if step % 16 == 0:
        real_images.requires_grad = True
        d_real = discriminator(real_images)
        r1_grads = torch.autograd.grad(d_real.sum(), real_images)[0]
        r1_penalty = r1_grads.square().sum(dim=[1,2,3]).mean()
        d_loss += 10.0 * r1_penalty

    d_optimizer.zero_grad()
    d_loss.backward()
    d_optimizer.step()

    # 2. Generator training
    z = torch.randn(batch_size, latent_dim)
    fake_images = generator(z)
    d_fake = discriminator(fake_images)
    g_loss = F.softplus(-d_fake).mean()

    g_optimizer.zero_grad()
    g_loss.backward()
    g_optimizer.step()

[VQ-VAE Architecture]

Input Image     Encoder      Quantization      Decoder      Reconstructed
  (256x256) --> [E(x)] --> z_e --> [Codebook] --> z_q --> [D(z_q)] --> Image
                             |        |
                             |   ┌────┴────┐
                             |   │ e_1     │
                             |   │ e_2     │  K code vectors
                             └──>│ ...     │  (Codebook)
                                 │ e_K     │
                                 └─────────┘

  z_q = e_k  where k = argmin_j ||z_e - e_j||
  (quantize to nearest code vector)

L = ||x - D(z_q)||²                    # Reconstruction Loss
  + ||sg[z_e] - e||²                   # Codebook Loss (EMA 업데이트로 대체 가능)
  + β * ||z_e - sg[e]||²              # Commitment Loss

- sg[·]: Stop-gradient 연산자
- β: Commitment loss 가중치 (보통 0.25)
- z_e: Encoder 출력
- e: 선택된 codebook 벡터

# VQ-VAE Codebook core training code
class VectorQuantizer(nn.Module):
    def __init__(self, num_embeddings, embedding_dim, commitment_cost=0.25):
        super().__init__()
        self.embedding = nn.Embedding(num_embeddings, embedding_dim)
        self.commitment_cost = commitment_cost

    def forward(self, z_e):
        # z_e: (B, D, H, W) -> (B*H*W, D)
        flat_z = z_e.permute(0, 2, 3, 1).reshape(-1, z_e.shape[1])

        # Find nearest codebook vector
        distances = (flat_z ** 2).sum(dim=1, keepdim=True) \
                  + (self.embedding.weight ** 2).sum(dim=1) \
                  - 2 * flat_z @ self.embedding.weight.t()
        indices = distances.argmin(dim=1)
        z_q = self.embedding(indices).view_as(z_e.permute(0, 2, 3, 1)).permute(0, 3, 1, 2)

        # Loss computation
        codebook_loss = F.mse_loss(z_q.detach(), z_e)      # commitment
        commitment_loss = F.mse_loss(z_q, z_e.detach())     # codebook
        loss = commitment_loss + self.commitment_cost * codebook_loss

        # Straight-Through Estimator
        z_q_st = z_e + (z_q - z_e).detach()
        return z_q_st, loss, indices

[VQ-VAE-2 Hierarchical Structure]

                    Top Level (작은 해상도)
                    ┌─────────────┐
                    │  32x32 grid │  Global structure info
                    │  Codebook   │  (composition, overall shape)
                    └──────┬──────┘
                           │
                    Bottom Level (큰 해상도)
                    ┌──────┴──────┐
                    │  64x64 grid │  Fine detail info
                    │  Codebook   │  (textures, edges)
                    └─────────────┘

q(x_t | x_{t-1}) = N(x_t; √(1-β_t) * x_{t-1}, β_t * I)

- Add a small amount of Gaussian noise at each timestep t
- β_t: noise schedule (보통 linear 또는 cosine)
- After T steps, x_T approximately equals N(0, I) (pure Gaussian noise)

q(x_t | x_0) = N(x_t; √(ᾱ_t) * x_0, (1-ᾱ_t) * I)

where ᾱ_t = ∏_{s=1}^{t} α_s,  α_t = 1 - β_t

=> x_t = √(ᾱ_t) * x_0 + √(1-ᾱ_t) * ε,  ε ~ N(0, I)

p_θ(x_{t-1} | x_t) = N(x_{t-1}; μ_θ(x_t, t), σ_t² * I)

- Neural network epsilon_theta predicts noise epsilon added to x_t
- Remove predicted noise to recover x_{t-1}

L_simple = E_{t, x_0, ε} [ ||ε - ε_θ(x_t, t)||² ]

- t ~ Uniform(1, T)
- ε ~ N(0, I)
- x_t = √(ᾱ_t) * x_0 + √(1-ᾱ_t) * ε

# DDPM core training loop
def train_step(model, x_0, noise_scheduler):
    batch_size = x_0.shape[0]

    # 1. Random timestep sampling
    t = torch.randint(0, num_timesteps, (batch_size,))

    # 2. Noise sampling
    noise = torch.randn_like(x_0)

    # 3. Forward process: generate x_t
    alpha_bar_t = noise_scheduler.alpha_bar[t]
    x_t = torch.sqrt(alpha_bar_t) * x_0 + torch.sqrt(1 - alpha_bar_t) * noise

    # 4. Predict noise
    noise_pred = model(x_t, t)

    # 5. Loss computation (MSE)
    loss = F.mse_loss(noise_pred, noise)

    return loss

# Cosine Schedule implementation
def cosine_beta_schedule(timesteps, s=0.008):
    steps = timesteps + 1
    x = torch.linspace(0, timesteps, steps)
    alphas_cumprod = torch.cos(((x / timesteps) + s) / (1 + s) * math.pi * 0.5) ** 2
    alphas_cumprod = alphas_cumprod / alphas_cumprod[0]
    betas = 1 - (alphas_cumprod[1:] / alphas_cumprod[:-1])
    return torch.clip(betas, 0.0001, 0.9999)

# EDM Noise Schedule (Karras et al., 2022)
def edm_sigma_schedule(num_steps, sigma_min=0.002, sigma_max=80.0, rho=7.0):
    step_indices = torch.arange(num_steps)
    t_steps = (sigma_max ** (1/rho) + step_indices / (num_steps - 1)
               * (sigma_min ** (1/rho) - sigma_max ** (1/rho))) ** rho
    return t_steps

[Latent Diffusion Model Architecture]

                          Text Prompt
                              │
                         ┌────┴────┐
                         │  CLIP   │
                         │ Encoder │
                         └────┬────┘
                              │ text embeddings
                              │
┌──────┐    ┌──────┐    ┌─────┴──────┐    ┌──────┐    ┌──────┐
│Image │    │ VAE  │    │   U-Net    │    │ VAE  │    │Output│
│(512  │--->│Encode│--->│ (Denoising │--->│Decode│--->│Image │
│x512) │    │  r   │    │  in Latent │    │  r   │    │(512  │
│      │    │      │    │   Space)   │    │      │    │x512) │
└──────┘    └──────┘    └────────────┘    └──────┘    └──────┘
              │                                 │
              │  64x64x4                        │
              │  (8x downsampling)              │
              └─────────────────────────────────┘
                    Latent Space (z)

Training: Diffusion in Latent Space
Inference: Random noise z_T -> U-Net Denoising -> VAE Decode -> Image

# Latent Diffusion core training
class LatentDiffusionTrainer:
    def __init__(self, vae, unet, text_encoder, noise_scheduler):
        self.vae = vae              # Frozen
        self.unet = unet            # Trainable
        self.text_encoder = text_encoder  # Frozen
        self.noise_scheduler = noise_scheduler

    def train_step(self, images, captions):
        # 1. Latent encoding with VAE (no gradient needed)
        with torch.no_grad():
            latents = self.vae.encode(images).latent_dist.sample()
            latents = latents * self.vae.config.scaling_factor  # 0.18215

        # 2. Text embedding (no gradient needed)
        with torch.no_grad():
            text_embeddings = self.text_encoder(captions)

        # 3. Add noise
        noise = torch.randn_like(latents)
        timesteps = torch.randint(0, 1000, (latents.shape[0],))
        noisy_latents = self.noise_scheduler.add_noise(latents, noise, timesteps)

        # 4. Predict noise
        noise_pred = self.unet(noisy_latents, timesteps, text_embeddings)

        # 5. MSE loss
        loss = F.mse_loss(noise_pred, noise)
        return loss

[U-Net with Cross-Attention Structure]

Input z_t ─────────────────────────────────────────── Output ε_θ
    │                                                     ▲
    ▼                                                     │
┌────────┐  ┌────────┐  ┌────────┐      ┌────────┐  ┌────────┐
│ Down   │  │ Down   │  │ Down   │      │  Up    │  │  Up    │
│ Block  │──│ Block  │──│ Block  │──┐   │ Block  │──│ Block  │
│ 64x64  │  │ 32x32  │  │ 16x16  │  │   │ 32x32  │  │ 64x64  │
└────────┘  └────────┘  └────────┘  │   └────────┘  └────────┘
    │            │            │     │        ▲           ▲
    │            │            │     ▼        │           │
    │            │            │  ┌────────┐  │           │
    │            │            └──│ Middle │──┘           │
    │            │               │ Block  │              │
    │            │               │ 16x16  │              │
    │            └───────────────└────────┘──────────────┘
    │                    (skip connections)
    └────────────────────────────────────────────────────┘

Inside each Block:
┌──────────────────────────────────────┐
│  ResNet Block                         │
│  ├── GroupNorm → SiLU → Conv         │
│  ├── Timestep Embedding injection          │
│  └── GroupNorm → SiLU → Conv         │
│                                       │
│  Self-Attention Block                 │
│  ├── LayerNorm → Self-Attention      │
│  └── Skip Connection                  │
│                                       │
│  Cross-Attention Block                │
│  ├── LayerNorm                        │
│  ├── Q = Linear(latent features)     │
│  ├── K = Linear(text embeddings)     │  ← Text Conditioning
│  ├── V = Linear(text embeddings)     │
│  └── Attention(Q, K, V)             │
│                                       │
│  Feed-Forward Block                   │
│  ├── LayerNorm → Linear → GEGLU     │
│  └── Linear → Skip Connection        │
└──────────────────────────────────────┘

학습 시:
  - probability p_uncond (예: 10%): ε_θ(x_t, t, ∅)  (unconditional)
  - probability 1-p_uncond:          ε_θ(x_t, t, c)  (conditional)

추론 시:
  ε_guided = ε_θ(x_t, t, ∅) + w * (ε_θ(x_t, t, c) - ε_θ(x_t, t, ∅))

  - w: guidance scale (보통 7.5 ~ 15)
  - w=1: conditional 예측 그대로
  - w>1: 텍스트 조건 방향으로 더 강하게 이동

# Classifier-Free Guidance training implementation
def train_step_cfg(model, x_0, text_cond, p_uncond=0.1):
    noise = torch.randn_like(x_0)
    t = torch.randint(0, T, (x_0.shape[0],))
    x_t = add_noise(x_0, noise, t)

    # Randomly drop conditioning
    mask = torch.rand(x_0.shape[0]) < p_uncond
    cond = text_cond.clone()
    cond[mask] = empty_text_embedding  # null conditioning

    noise_pred = model(x_t, t, cond)
    loss = F.mse_loss(noise_pred, noise)
    return loss

# Classifier-Free Guidance inference
def sample_cfg(model, x_T, text_cond, guidance_scale=7.5):
    x_t = x_T
    for t in reversed(range(T)):
        # Unconditional prediction
        eps_uncond = model(x_t, t, empty_text_embedding)
        # Conditional prediction
        eps_cond = model(x_t, t, text_cond)
        # Guided prediction
        eps = eps_uncond + guidance_scale * (eps_cond - eps_uncond)
        x_t = denoise_step(x_t, eps, t)
    return x_t

[DALL-E 2 Training Pipeline]

Text ──→ CLIP Text Encoder ──→ text embedding
                                    │
                              ┌─────┴─────┐
                              │   Prior    │  text emb → CLIP image emb
                              │ (Diffusion)│
                              └─────┬─────┘
                                    │ CLIP image embedding
                              ┌─────┴─────┐
                              │  Decoder   │  CLIP image emb → 64x64 image
                              │ (Diffusion)│
                              └─────┬─────┘
                                    │ 64x64
                              ┌─────┴─────┐
                              │ Super-Res  │  64x64 → 256x256 → 1024x1024
                              │ (Diffusion)│
                              └─────────── ┘

[Imagen Architecture]

Text ──→ T5-XXL (frozen) ──→ text embeddings
                                    │
                              ┌─────┴─────┐
                              │ Base Model │  64x64 생성
                              │  (U-Net)   │  cross-attention
                              └─────┬─────┘
                                    │
                              ┌─────┴─────┐
                              │ SR Model 1 │  64x64 → 256x256
                              │  (U-Net)   │
                              └─────┬─────┘
                                    │
                              ┌─────┴─────┐
                              │ SR Model 2 │  256x256 → 1024x1024
                              │  (U-Net)   │
                              └─────────── ┘

[DiT Block Structure]

     Input Tokens (patchified latent)
           │
    ┌──────┴──────┐
    │  LayerNorm  │ ← adaLN-Zero (adaptive)
    │  (adaptive) │   γ, β = MLP(timestep + class)
    └──────┬──────┘
           │
    ┌──────┴──────┐
    │    Self-     │
    │  Attention   │
    └──────┬──────┘
           │ (+ residual)
    ┌──────┴──────┐
    │  LayerNorm  │ ← adaLN-Zero
    │  (adaptive) │
    └──────┬──────┘
           │
    ┌──────┴──────┐
    │ Pointwise   │
    │    FFN      │
    └──────┬──────┘
           │ (+ residual, scaled by α)
           ▼
     Output Tokens

[DALL-E Training Pipeline]

Stage 1: dVAE 학습
  Image (256x256) ──→ dVAE Encoder ──→ 32x32 grid of tokens (8192 vocabulary)
                                           ──→ dVAE Decoder ──→ Reconstructed Image

  Loss: Reconstruction + KL Divergence (Gumbel-Softmax relaxation)

Stage 2: Autoregressive Transformer 학습
  [BPE text tokens (256)] + [Image tokens (1024)] = 1280 tokens

  Transformer (12B params):
  - 64 layers, 62 attention heads
  - 학습 목적: next-token prediction (cross-entropy)
  - 텍스트 토큰은 causal attention (좌→우)
  - 이미지 토큰은 row-major order로 자기회귀 생성
  - 텍스트→이미지 cross-attention 포함

[Diffusion vs Flow Matching Comparison]

Diffusion (Stochastic):              Flow Matching (Deterministic):
  x_0 ~~~> x_T                         x_0 ──────> x_1
  (curved path, requires many steps)              (straight path, fewer steps possible)

  x₀ •                                x₀ •
     \  Curved                            \  Straight
      \    path                            \    path
       \                                    \
        \                                    \
     x_T •                              x₁ •  (= noise)

dx = f(x,t)dt + g(t)dW              dx/dt = v_θ(x_t, t)
(SDE 기반)                           (ODE 기반, velocity field 학습)

L_FM = E_{t, x_0, x_1} [ ||v_θ(x_t, t) - u_t(x_t | x_0, x_1)||² ]

where:
  x_t = (1 - t) * x_0 + t * x_1      (linear interpolation)
  u_t = x_1 - x_0                      (target velocity: 직선 path)
  t ~ Uniform(0, 1)                    (또는 logit-normal)
  x_0 ~ p_data (실제 데이터)
  x_1 ~ N(0, I) (가우시안 노이즈)

# Rectified Flow core training
def rectified_flow_train_step(model, x_0, x_1=None):
    """
    x_0: 실제 데이터 (latent)
    x_1: 노이즈 (None이면 랜덤 샘플링)
    """
    if x_1 is None:
        x_1 = torch.randn_like(x_0)

    # Time sampling (logit-normal for SD3)
    t = torch.sigmoid(torch.randn(x_0.shape[0]))  # logit-normal
    t = t.view(-1, 1, 1, 1)

    # Linear interpolation
    x_t = (1 - t) * x_0 + t * x_1

    # Target velocity (straight direction)
    target_v = x_1 - x_0

    # Velocity prediction
    v_pred = model(x_t, t)

    # Loss
    loss = F.mse_loss(v_pred, target_v)
    return loss

# SD3's Logit-Normal Timestep Sampling
def logit_normal_sampling(batch_size, m=0.0, s=1.0):
    """Give more weight to middle timesteps"""
    u = torch.randn(batch_size) * s + m
    t = torch.sigmoid(u)  # (0, 1) 범위
    return t

[MM-DiT Block]

  Image Tokens          Text Tokens
       │                     │
  ┌────┴────┐          ┌────┴────┐
  │adaLN(t) │          │adaLN(t) │
  └────┬────┘          └────┬────┘
       │                     │
       └──────┬──────────────┘
              │ (concatenate)
       ┌──────┴──────┐
       │ Joint Self- │  Image-text tokens
       │  Attention  │     attend to each other
       └──────┬──────┘
              │ (split)
       ┌──────┴──────────────┐
       │                     │
  ┌────┴────┐          ┌────┴────┐
  │   FFN   │          │   FFN   │
  │ (image) │          │ (text)  │
  └────┬────┘          └────┬────┘
       │                     │
  Image Out             Text Out

[CLIP Training Process]

  Image ──→ Image Encoder ──→ image embedding ─┐
                                                 ├─ cosine similarity
  Text  ──→ Text Encoder  ──→ text embedding  ─┘

  Training objective: Increase similarity for matching pairs, decrease for non-matching pairs
  (InfoNCE Loss)

# Cross-Attention implementation
class CrossAttention(nn.Module):
    def __init__(self, d_model, d_context, n_heads):
        super().__init__()
        self.n_heads = n_heads
        self.d_head = d_model // n_heads

        self.to_q = nn.Linear(d_model, d_model, bias=False)      # latent → Q
        self.to_k = nn.Linear(d_context, d_model, bias=False)     # text → K
        self.to_v = nn.Linear(d_context, d_model, bias=False)     # text → V
        self.to_out = nn.Linear(d_model, d_model)

    def forward(self, x, context):
        """
        x: (B, H*W, d_model) - 이미지 latent features
        context: (B, seq_len, d_context) - 텍스트 임베딩
        """
        q = self.to_q(x)          # 이미지가 Query
        k = self.to_k(context)    # 텍스트가 Key
        v = self.to_v(context)    # 텍스트가 Value

        # Multi-head reshape
        q = q.view(B, -1, self.n_heads, self.d_head).transpose(1, 2)
        k = k.view(B, -1, self.n_heads, self.d_head).transpose(1, 2)
        v = v.view(B, -1, self.n_heads, self.d_head).transpose(1, 2)

        # Attention
        attn = (q @ k.transpose(-2, -1)) / math.sqrt(self.d_head)
        attn = F.softmax(attn, dim=-1)
        out = attn @ v

        out = out.transpose(1, 2).reshape(B, -1, d_model)
        return self.to_out(out)

[Text Embedding Types]

Text: "a photo of a cat"
         │
    ┌────┴────┐
    │  Text   │
    │ Encoder │
    └────┬────┘
         │
    ┌────┴──────────────────────┐
    │                           │
    ▼                           ▼
 Sequence Embeddings         Pooled Embedding
 (token-level)               (sentence-level)
 [h_1, h_2, ..., h_n]       h_pool = h_[EOS]
 Shape: (seq_len, d)         Shape: (d,)
    │                           │
    │                           │
    ▼                           ▼
 Cross-Attention에 사용       Global conditioning에 사용
 (세밀한 토큰별 정보)          (전체 문장 의미)
                              - Timestep embedding에 더하기
                              - adaLN 파라미터 조절
                              - Vector conditioning

# SDXL Text Conditioning
def get_sdxl_text_embeddings(text, clip_l, clip_g):
    # CLIP ViT-L: sequence embeddings (77, 768)
    clip_l_output = clip_l(text)
    clip_l_seq = clip_l_output.last_hidden_state      # (77, 768)
    clip_l_pooled = clip_l_output.pooler_output        # (768,)

    # OpenCLIP ViT-bigG: sequence embeddings (77, 1280)
    clip_g_output = clip_g(text)
    clip_g_seq = clip_g_output.last_hidden_state       # (77, 1280)
    clip_g_pooled = clip_g_output.pooler_output        # (1280,)

    # Concatenate sequence embeddings -> used for Cross-Attention
    text_embeddings = torch.cat([clip_l_seq, clip_g_seq], dim=-1)  # (77, 2048)

    # Concatenate pooled embeddings -> used for Vector conditioning
    pooled_embeddings = torch.cat([clip_l_pooled, clip_g_pooled], dim=-1)  # (2048,)

    return text_embeddings, pooled_embeddings

[LAION-5B Data Collection and Filtering Pipeline]

Common Crawl (웹 아카이브)
        │
        ▼
1. HTML 파싱: <img> 태그에서 src URL + alt-text 추출
        │
        ▼
2. 이미지 다운로드 (img2dataset)
   - 최소 해상도 필터: width, height ≥ 64
   - 최대 종횡비: 3:1
        │
        ▼
3. CLIP 유사도 필터링
   - OpenAI CLIP ViT-B/32로 image-text similarity 계산
   - 영어: cosine similarity ≥ 0.28
   - 기타 언어: cosine similarity ≥ 0.26
        │
        ▼
4. 안전성 필터링
   - NSFW 탐지 점수 (CLIP 기반)
   - Watermark 탐지 점수
   - Toxic content 탐지
        │
        ▼
5. 중복 제거 (deduplication)
   - 해시 기반 exact duplicate 제거
   - CLIP embedding 기반 near-duplicate 제거
        │
        ▼
최종: 58.5억 이미지-텍스트 pairs
 - 23.2억 영어
 - 22.6억 기타 100+ 언어
 - 12.7억 언어 미확인

# CLIP Score computation
from transformers import CLIPModel, CLIPProcessor
model = CLIPModel.from_pretrained("openai/clip-vit-large-patch14")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-large-patch14")

inputs = processor(text=[caption], images=[image], return_tensors="pt")
outputs = model(**inputs)
clip_score = outputs.logits_per_image.item() / 100.0  # normalized

[DALL-E 3 Caption Improvement Effect]

Before: "cat on table"
      -> Vague and lacks detail

After: "A fluffy orange tabby cat sitting on a round wooden
       dining table, natural sunlight streaming through a
       window behind, casting soft shadows. The cat has
       bright green eyes and is looking directly at the camera."
      -> Includes detailed attributes, spatial relationships, and lighting information

[LoRA Principle]

원본 가중치:  W_0 ∈ R^{d×k}  (고정, frozen)
LoRA 업데이트: ΔW = B × A      where A ∈ R^{r×k}, B ∈ R^{d×r}

최종 출력: h = W_0 x + ΔW x = W_0 x + B(Ax)

- r << min(d, k): low-rank (보통 4, 8, 16, 32, 64)
- 학습 파라미터: A와 B만 (전체 대비 매우 적음)
- 원본 가중치는 고정 → 메모리 효율적

# LoRA application example (Stable Diffusion U-Net attention layer)
class LoRALinear(nn.Module):
    def __init__(self, original_layer, rank=4, alpha=1.0):
        super().__init__()
        self.original = original_layer  # frozen
        in_features = original_layer.in_features
        out_features = original_layer.out_features

        # LoRA layers
        self.lora_A = nn.Linear(in_features, rank, bias=False)
        self.lora_B = nn.Linear(rank, out_features, bias=False)
        self.scale = alpha / rank

        # Initialization
        nn.init.kaiming_uniform_(self.lora_A.weight)
        nn.init.zeros_(self.lora_B.weight)  # Initialize B to 0 -> identical to original at start

    def forward(self, x):
        original_out = self.original(x)        # Frozen original output
        lora_out = self.lora_B(self.lora_A(x)) # LoRA update
        return original_out + self.scale * lora_out

# Diffusers LoRA training execution example
accelerate launch train_text_to_image_lora.py \
  --pretrained_model_name_or_path="stabilityai/stable-diffusion-xl-base-1.0" \
  --dataset_name="lambdalabs/naruto-blip-captions" \
  --resolution=1024 \
  --train_batch_size=1 \
  --gradient_accumulation_steps=4 \
  --max_train_steps=15000 \
  --learning_rate=1e-04 \
  --lr_scheduler="cosine" \
  --lr_warmup_steps=0 \
  --rank=4 \
  --mixed_precision="fp16" \
  --output_dir="./sdxl-naruto-lora"

[DreamBooth Training Process]

Input: 3-5 images of a specific subject + unique identifier "[V]"
      Example: "a [V] dog" (specific dog)

Training strategy:
1. Fine-tune model with subject images
   - "a [V] dog" → 해당 강아지 이미지

2. Prior Preservation Loss (Key!)
   - Pre-generate "a dog" images with the original model
   - Preserve general dog generation capability during fine-tuning
   - Prevent language drift

L = L_recon([V] images) + λ * L_prior(class images)

# DreamBooth + LoRA training (recommended combination)
# Based on diffusers library
accelerate launch train_dreambooth_lora.py \
  --pretrained_model_name_or_path="stabilityai/stable-diffusion-xl-base-1.0" \
  --instance_data_dir="./my_dog_images" \
  --instance_prompt="a photo of sks dog" \
  --class_data_dir="./class_dog_images" \
  --class_prompt="a photo of dog" \
  --with_prior_preservation \
  --prior_loss_weight=1.0 \
  --num_class_images=200 \
  --resolution=1024 \
  --train_batch_size=1 \
  --gradient_accumulation_steps=1 \
  --learning_rate=1e-4 \
  --lr_scheduler="constant" \
  --max_train_steps=500 \
  --rank=4 \
  --mixed_precision="fp16"

[Textual Inversion]

Existing token space:  [cat] [dog] [car] [tree] ...
                                  │
Add new token:               [S*] New concept to learn
                                  │
Training: Optimize only the embedding vector of [S*] with 3-5 images
Entire rest of model is frozen

Advantage: Minimal parameters (1 token = 768 or 1024 floats)
Disadvantage: Less expressive than LoRA/DreamBooth

[ControlNet Architecture]

                   Control Input (예: Canny edge)
                          │
                    ┌─────┴─────┐
                    │  Zero     │
                    │  Conv     │
                    └─────┬─────┘
                          │
                    ┌─────┴─────┐
Locked U-Net        │ Trainable │  Copy of U-Net Encoder
(원본 고정)          │  Copy of  │     (trainable)
    │               │ U-Net Enc │
    │               └─────┬─────┘
    │                     │
    │               ┌─────┴─────┐
    │               │  Zero     │  Output is 0 at training start
    │               │  Conv     │     (starts without affecting original model)
    │               └─────┬─────┘
    │                     │
    └─────── + ───────────┘  Add to original U-Net features
                  │
              Final Output

# Zero Convolution: Initialize weights and biases to 0
class ZeroConv(nn.Module):
    def __init__(self, in_channels, out_channels):
        super().__init__()
        self.conv = nn.Conv2d(in_channels, out_channels, 1)
        nn.init.zeros_(self.conv.weight)
        nn.init.zeros_(self.conv.bias)

    def forward(self, x):
        return self.conv(x)

# Training start: zero conv output = 0
# -> Adding ControlNet doesn't affect original model output
# -> Gradually reflects control signal as training progresses

[IP-Adapter Architecture]

Reference Image ──→ CLIP Image Encoder ──→ image features
                                               │
                                         ┌─────┴─────┐
                                         │ Projection │  Trainable
                                         │   Layer    │
                                         └─────┬─────┘
                                               │
                                         ┌─────┴─────┐
                                         │ Decoupled │  Separate cross-attention
                                         │ Cross-Attn │    (separated from text cross-attn)
                                         └─────┬─────┘
                                               │
Original U-Net Cross-Attention ────── + ───────┘
(text conditioning)

출력 = Text_CrossAttn(Q, K_text, V_text) + λ * Image_CrossAttn(Q, K_img, V_img)

[Consistency Models Key Idea]

Diffusion: x_T → x_{T-1} → ... → x_1 → x_0  (hundreds of steps)

Consistency:
  PF-ODE trajectory 위의 모든 점 x_t가
  동일한 x_0로 매핑되도록 학습

  f_θ(x_t, t) = x_0  ∀t ∈ [0, T]

  Key 제약: f_θ(x_0, 0) = x_0 (self-consistency)

     x_T ────→ f_θ ────→ x_0
      │                    ↑
     x_t ────→ f_θ ───────┘  (maps to the same x_0!)
      │                    ↑
    x_t' ────→ f_θ ───────┘

[PixArt-alpha 3-Stage Training]

Stage 1: Pixel Dependency 학습 (저비용)
  - ImageNet 사전학습된 DiT에서 시작
  - 클래스 조건부 → T2I 전환의 기초

Stage 2: Text-Image Alignment 학습
  - Cross-attention으로 텍스트 조건 주입
  - LLaVA로 생성한 고품질 synthetic caption 사용

Stage 3: High-quality Aesthetic 학습
  - 고품질 미적 Dataset으로 fine-tuning
  - JourneyDB 등 활용

총 학습 비용: ~675 A100 GPU days
(SD 1.5의 ~6,250 A100 GPU days 대비 10.8%)

[Stable Diffusion Lineage by Generation]

SD 1.x (2022)     SDXL (2023)       SD3 (2024)         Flux (2024)
    │                  │                │                   │
  U-Net 860M       U-Net 2.6B      MM-DiT 2-8B        MM-DiT ~12B
    │                  │                │                   │
  CLIP ViT-L       CLIP-L +          CLIP-L +            CLIP-L +
                   OpenCLIP-G        OpenCLIP-G +        OpenCLIP-G +
                                     T5-XXL               T5-XXL
    │                  │                │                   │
  Diffusion        Diffusion        Rectified            Rectified
  (DDPM)           (DDPM)           Flow                 Flow
    │                  │                │                   │
  512x512          1024x1024        1024x1024            1024x1024+
    │                  │                │                   │
  CFG 7.5          CFG 5-9          CFG 3.5-7            Guidance
                                                         Distillation

# EDM Framework (Karras et al., 2022)
# σ(t) 기반 noise schedule - Zero Terminal SNR 보장
# 기존 SD의 linear schedule 대비 색상/대비 크게 개선

def edm_precondition(sigma, x_noisy, F_theta):
    """EDM Preconditioning"""
    c_skip = 1.0 / (sigma ** 2 + 1)
    c_out = sigma / (sigma ** 2 + 1).sqrt()
    c_in = 1.0 / (sigma ** 2 + 1).sqrt()
    c_noise = sigma.log() / 4

    D_x = c_skip * x_noisy + c_out * F_theta(c_in * x_noisy, c_noise)
    return D_x

[Large-Scale Distributed Training Configuration]

┌─────────────────────────────────────────────────────┐
│                  Data Parallel (DP/DDP)               │
│                                                       │
│  GPU 0        GPU 1        GPU 2        GPU 3        │
│  ┌──────┐    ┌──────┐    ┌──────┐    ┌──────┐       │
│  │Full  │    │Full  │    │Full  │    │Full  │       │
│  │Model │    │Model │    │Model │    │Model │       │
│  │Copy  │    │Copy  │    │Copy  │    │Copy  │       │
│  └──────┘    └──────┘    └──────┘    └──────┘       │
│  Batch 1     Batch 2     Batch 3     Batch 4        │
│                                                       │
│  -> Synchronize gradients with All-Reduce                       │
│  -> Different data batches on each GPU                       │
└─────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────┐
│              FSDP (Fully Sharded Data Parallel)       │
│                                                       │
│  GPU 0        GPU 1        GPU 2        GPU 3        │
│  ┌──────┐    ┌──────┐    ┌──────┐    ┌──────┐       │
│  │Shard │    │Shard │    │Shard │    │Shard │       │
│  │ 1/4  │    │ 2/4  │    │ 3/4  │    │ 4/4  │       │
│  └──────┘    └──────┘    └──────┘    └──────┘       │
│                                                       │
│  -> Shard model parameters across GPUs                   │
│  -> All-Gather only needed shards during Forward/Backward       │
│  -> Maximize memory efficiency (enables 8B+ model training)               │
└─────────────────────────────────────────────────────┘

# Diffusers-based Text-to-Image full training pipeline
from diffusers import AutoencoderKL, UNet2DConditionModel, DDPMScheduler
from transformers import CLIPTextModel, CLIPTokenizer
from accelerate import Accelerator
import torch

# 1. Load models
vae = AutoencoderKL.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0", subfolder="vae"
)
unet = UNet2DConditionModel.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0", subfolder="unet"
)
text_encoder = CLIPTextModel.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0", subfolder="text_encoder"
)
tokenizer = CLIPTokenizer.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0", subfolder="tokenizer"
)
noise_scheduler = DDPMScheduler.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0", subfolder="scheduler"
)

# 2. Freeze VAE and Text Encoder
vae.requires_grad_(False)
text_encoder.requires_grad_(False)

# 3. Accelerator setup (distributed training + Mixed Precision)
accelerator = Accelerator(
    mixed_precision="fp16",          # or "bf16"
    gradient_accumulation_steps=4,
)

# 4. Optimizer
optimizer = torch.optim.AdamW(
    unet.parameters(),
    lr=1e-4,
    betas=(0.9, 0.999),
    weight_decay=1e-2,
    eps=1e-8,
)

# 5. EMA setup
from diffusers.training_utils import EMAModel
ema_unet = EMAModel(
    unet.parameters(),
    decay=0.9999,
    use_ema_warmup=True,
)

# 6. Prepare for distributed training
unet, optimizer, dataloader = accelerator.prepare(unet, optimizer, dataloader)

# 7. Training loop
for epoch in range(num_epochs):
    for batch in dataloader:
        with accelerator.accumulate(unet):
            images = batch["images"]
            captions = batch["captions"]

            # Latent encoding
            with torch.no_grad():
                latents = vae.encode(images).latent_dist.sample()
                latents = latents * vae.config.scaling_factor

            # Text encoding
            with torch.no_grad():
                text_inputs = tokenizer(captions, padding="max_length",
                                       max_length=77, return_tensors="pt")
                text_embeds = text_encoder(text_inputs.input_ids)[0]

            # Add noise
            noise = torch.randn_like(latents)
            timesteps = torch.randint(0, 1000, (latents.shape[0],))
            noisy_latents = noise_scheduler.add_noise(latents, noise, timesteps)

            # Classifier-Free Guidance: random caption dropout
            if torch.rand(1) < 0.1:  # 10% probability로 unconditional
                text_embeds = torch.zeros_like(text_embeds)

            # Predict noise
            noise_pred = unet(noisy_latents, timesteps, text_embeds).sample

            # Loss computation
            loss = F.mse_loss(noise_pred, noise)

            # Backward
            accelerator.backward(loss)
            accelerator.clip_grad_norm_(unet.parameters(), 1.0)
            optimizer.step()
            optimizer.zero_grad()

            # EMA update
            ema_unet.step(unet.parameters())

[Mixed Precision Training]

Forward Pass:
  - Model weights: FP16/BF16 (half memory)
  - Activation: FP16/BF16

Loss Scaling:
  - Multiply loss by a large scale (e.g., 2^16) to prevent gradient underflow
  - Scale down gradient again after backward

Backward Pass:
  - Gradient: FP16/BF16

Optimizer Step:
  - Master Weights: FP32 (maintain precision!)
  - Update FP32 master weights then create FP16 copy

# BF16 Mixed Precision config (accelerate-based)
# accelerate config (YAML)
compute_environment: LOCAL_MACHINE
distributed_type: MULTI_GPU
mixed_precision: bf16
num_machines: 1
num_processes: 8

[EMA Update]

θ_ema ← λ * θ_ema + (1 - λ) * θ_model

- λ: decay rate (보통 0.9999 ~ 0.99999)
- θ_model: 현재 학습 중인 모델 가중치
- θ_ema: EMA 가중치 (추론 시 사용)
- Effect: gradient noise를 평활화하여 더 안정적인 가중치

# Diffusers EMA implementation
from diffusers.training_utils import EMAModel

# Create EMA model
ema_model = EMAModel(
    unet.parameters(),
    decay=0.9999,              # decay rate
    use_ema_warmup=True,       # warmup 사용
    inv_gamma=1.0,             # warmup 파라미터
    power=3/4,                 # warmup 파라미터
)

# Update at every training step
ema_model.step(unet.parameters())

# Apply EMA weights at inference
ema_model.copy_to(unet.parameters())

# Or use context manager
with ema_model.average_parameters():
    # EMA weights are used inside this block
    output = unet(noisy_latents, timesteps, text_embeds)

[T2I Training Methodology Evolution]

Efficiency:  Full Training ──→ LoRA/Adapter ──→ Prompt Tuning
         (months, $1M+)     (minutes, less than $1)      (seconds)

Architecture: U-Net ────────→ DiT ─────────→ MM-DiT + Flow Matching
          (SD 1.x-SDXL)    (DiT, PixArt)   (SD3, Flux)

Generation speed: 50-1000 steps ──→ 20-50 steps ──→ 1-4 steps
           (DDPM)            (DDIM, DPM++)   (LCM, LADD, CM)

Data quality: Web crawling ──→ 필터링 ──→ Synthetic Caption
            (LAION raw)    (aesthetic)  (DALL-E 3 방식)

Text understanding: CLIP only ──→ CLIP + T5 ──→ 3중 Encoder
             (SD 1.x)     (Imagen)      (SD3, Flux)