0%
💡 왼쪽 원문을 읽으면서 오른쪽에 따라 써보세요. Tab 키로 힌트를 받을 수 있습니다.
引言
生成式AI(Generative AI)是当今技术领域最炙手可热的方向。DALL-E 3 仅凭一行文字就能生成照片般逼真的图像，Stable Diffusion 能创作艺术作品，Sora 则能生成视频。
这一切创新的背后，是数十年间不断演进的生成模型。从 VAE(变分自编码器)、GAN(生成对抗网络)，到如今占据主导地位的扩散模型 — 让我们彻底掌握它们各自的思路、数学原理，以及具体的实现方法。
1. 生成模型(Generative Model)概述
1.1 生成模型 vs 判别模型
深度学习模型大致可分为两类。
判别模型(Discriminative Model)：在给定输入数据 x 的条件下，学习标签 y 的条件概率 P(y|x)。图像分类、目标检测等任务属于此类。
生成模型(Generative Model)：学习数据的联合概率分布 P(x) 或 P(x, y)。训练完成后可以生成新的数据样本。
生成模型的核心问题是："这份数据是如何生成的？如何从同一分布中生成新的样本？"
1.2 潜在空间(Latent Space)的概念
大多数生成模型都会利用潜在空间(Latent Space)这一低维表示空间。
例如，28x28 的 MNIST 图像(784 维)可以压缩为 2 到 100 维的潜在向量。在这个潜在空间中：
- 数字"3"和数字"8"彼此相邻
- 对中间点的向量解码，会得到介于"3"和"8"之间的形态
- 在潜在空间中游走(插值)可以实现连续的变换
1.3 生成模型的应用领域
- 图像生成：生成逼真的人脸、风景、艺术作品
- 图像到图像转换：白天照片 → 夜晚照片，草图 → 照片
- 数据增强：补充不足的训练数据
- 药物发现：生成新的分子结构
- 文本生成：GPT 系列语言模型
- 音乐/语音合成：创作新音乐、TTS
2. 自编码器(Autoencoder)
我们从理解 VAE 所需的基础 — 自编码器开始讲起。
2.1 编码器-解码器结构
自编码器由两部分组成。
- 编码器(Encoder)：高维输入 x → 低维潜在向量 z
- 解码器(Decoder)：低维潜在向量 z → 重构输出 x'
目标：训练使 x' 尽可能接近 x(即最小化重构损失)。
import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import datasets, transforms
from torch.utils.data import DataLoader
class Autoencoder(nn.Module):
"""基础自编码器（用于 MNIST）"""
def __init__(self, input_dim=784, latent_dim=32):
super().__init__()
# 编码器
self.encoder = nn.Sequential(
nn.Linear(input_dim, 256),
nn.ReLU(),
nn.Linear(256, 128),
nn.ReLU(),
nn.Linear(128, latent_dim),
nn.ReLU()
)
# 解码器
self.decoder = nn.Sequential(
nn.Linear(latent_dim, 128),
nn.ReLU(),
nn.Linear(128, 256),
nn.ReLU(),
nn.Linear(256, input_dim),
nn.Sigmoid() # 将像素值限制在 [0, 1] 范围内
)
def forward(self, x):
# 展平
x = x.view(x.size(0), -1)
z = self.encoder(x)
x_reconstructed = self.decoder(z)
return x_reconstructed, z
def encode(self, x):
x = x.view(x.size(0), -1)
return self.encoder(x)
def decode(self, z):
return self.decoder(z).view(-1, 1, 28, 28)
def train_autoencoder(epochs=20):
"""训练自编码器"""
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
# MNIST 数据集
transform = transforms.Compose([
transforms.ToTensor(),
])
train_dataset = datasets.MNIST('./data', train=True, download=True, transform=transform)
train_loader = DataLoader(train_dataset, batch_size=128, shuffle=True)
model = Autoencoder().to(device)
optimizer = optim.Adam(model.parameters(), lr=1e-3)
criterion = nn.BCELoss() # 二元交叉熵
for epoch in range(epochs):
total_loss = 0
for batch_idx, (data, _) in enumerate(train_loader):
data = data.to(device)
optimizer.zero_grad()
reconstructed, z = model(data)
target = data.view(data.size(0), -1)
loss = criterion(reconstructed, target)
loss.backward()
optimizer.step()
total_loss += loss.item()
avg_loss = total_loss / len(train_loader)
print(f"Epoch {epoch+1}/{epochs}, Loss: {avg_loss:.4f}")
return model
2.2 自编码器的局限
基础自编码器能很好地压缩数据，但生成新样本却很困难。潜在空间并不规则，从任意潜在向量解码，往往会得到毫无意义的图像。
VAE 正是为了解决这个问题而诞生的。
3. 变分自编码器(VAE)
VAE(Variational Autoencoder)由 Kingma 与 Welling 在 2013 年发表的开创性论文(arXiv:1312.6114)中提出。
3.1 VAE 的核心思路
VAE 的核心，是在潜在空间中学习一个概率分布。
- 基础自编码器：z = encoder(x)(确定性的点)
- VAE：z ~ N(μ, σ²)(从高斯分布中采样)
编码器现在输出的不再是潜在向量 z 本身，而是分布的参数(均值 μ、方差 σ²)。
生成新图像时，从标准正态分布 N(0, I)中采样 z，输入解码器即可。
3.2 ELBO(证据下界)
VAE 的训练目标是最大化数据的对数似然 log P(x)。由于这一目标难以直接优化，因此转而采用最大化 ELBO 的方式。
log P(x) >= E_q[log P(x|z)] - KL[q(z|x) || P(z)]
↑ ↑
重构损失 KL 散度
ELBO = 重构损失 + KL 散度
- 重构损失：编码后再解码得到的结果，与原始输入有多相似
- KL 散度：学习到的潜在分布 q(z|x)，与先验分布 P(z) = N(0, I)有多接近
3.3 重参数化技巧(Reparameterization Trick)
若直接从 z ~ N(μ, σ²)中采样，反向传播将无法进行。重参数化技巧正是为了解决这一问题。
z = μ + σ * ε, ε ~ N(0, I)
这样一来，随机性(ε)就与网络参数分离开来，从而可以计算关于 μ 和 σ 的梯度。
3.4 完整的 VAE 实现(MNIST)
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torchvision import datasets, transforms
from torch.utils.data import DataLoader
import matplotlib.pyplot as plt
import numpy as np
class VAE(nn.Module):
"""变分自编码器"""
def __init__(self, input_dim=784, hidden_dim=400, latent_dim=20):
super(VAE, self).__init__()
self.latent_dim = latent_dim
# 编码器：输入 -> μ, log(σ²)
self.fc1 = nn.Linear(input_dim, hidden_dim)
self.fc_mu = nn.Linear(hidden_dim, latent_dim) # 均值
self.fc_logvar = nn.Linear(hidden_dim, latent_dim) # log 方差
# 解码器：潜在向量 -> 重构
self.fc3 = nn.Linear(latent_dim, hidden_dim)
self.fc4 = nn.Linear(hidden_dim, input_dim)
def encode(self, x):
"""编码器：x -> (μ, log_var)"""
h = F.relu(self.fc1(x))
return self.fc_mu(h), self.fc_logvar(h)
def reparameterize(self, mu, logvar):
"""重参数化技巧：z = μ + ε * σ"""
if self.training:
std = torch.exp(0.5 * logvar) # σ = exp(0.5 * log σ²)
eps = torch.randn_like(std) # ε ~ N(0, I)
return mu + eps * std
else:
return mu # 推理时只使用均值
def decode(self, z):
"""解码器：z -> x'"""
h = F.relu(self.fc3(z))
return torch.sigmoid(self.fc4(h))
def forward(self, x):
x_flat = x.view(-1, 784)
mu, logvar = self.encode(x_flat)
z = self.reparameterize(mu, logvar)
x_recon = self.decode(z)
return x_recon, mu, logvar
def generate(self, num_samples, device):
"""从标准正态分布采样以生成图像"""
with torch.no_grad():
z = torch.randn(num_samples, self.latent_dim).to(device)
samples = self.decode(z)
return samples.view(num_samples, 1, 28, 28)
def vae_loss(x_recon, x, mu, logvar, beta=1.0):
"""
VAE 损失 = 重构损失 + β * KL 散度
beta=1：标准 VAE
beta>1：β-VAE（潜在表示的解耦程度更高）
"""
# 重构损失（BCE）
recon_loss = F.binary_cross_entropy(
x_recon, x.view(-1, 784),
reduction='sum'
)
# KL 散度：KL[N(μ, σ²) || N(0, 1)]
# = -0.5 * Σ(1 + log σ² - μ² - σ²)
kl_loss = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())
return recon_loss + beta * kl_loss
def train_vae(epochs=50, latent_dim=20, beta=1.0):
"""训练 VAE"""
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
transform = transforms.Compose([transforms.ToTensor()])
train_dataset = datasets.MNIST('./data', train=True, download=True, transform=transform)
train_loader = DataLoader(train_dataset, batch_size=128, shuffle=True)
test_dataset = datasets.MNIST('./data', train=False, transform=transform)
test_loader = DataLoader(test_dataset, batch_size=128)
model = VAE(latent_dim=latent_dim).to(device)
optimizer = optim.Adam(model.parameters(), lr=1e-3)
train_losses = []
for epoch in range(epochs):
model.train()
total_loss = 0
for data, _ in train_loader:
data = data.to(device)
optimizer.zero_grad()
x_recon, mu, logvar = model(data)
loss = vae_loss(x_recon, data, mu, logvar, beta)
loss.backward()
optimizer.step()
total_loss += loss.item()
avg_loss = total_loss / len(train_dataset)
train_losses.append(avg_loss)
if (epoch + 1) % 10 == 0:
print(f"Epoch {epoch+1}/{epochs}, Loss: {avg_loss:.4f}")
return model, train_losses
def interpolate_latent_space(model, img1, img2, steps=10, device='cpu'):
"""在潜在空间中对两张图像进行插值"""
model.eval()
with torch.no_grad():
# 编码潜在向量
z1_flat = img1.view(-1, 784).to(device)
z2_flat = img2.view(-1, 784).to(device)
mu1, _ = model.encode(z1_flat)
mu2, _ = model.encode(z2_flat)
# 线性插值
interpolated_images = []
for alpha in np.linspace(0, 1, steps):
z_interp = (1 - alpha) * mu1 + alpha * mu2
img_recon = model.decode(z_interp)
interpolated_images.append(img_recon.view(1, 28, 28))
return interpolated_images
3.5 卷积 VAE(用于 CIFAR-10)
class ConvVAE(nn.Module):
"""卷积 VAE（用于彩色图像）"""
def __init__(self, latent_dim=128):
super().__init__()
self.latent_dim = latent_dim
# 卷积编码器
self.encoder_conv = nn.Sequential(
nn.Conv2d(3, 32, 4, stride=2, padding=1), # 16x16
nn.ReLU(),
nn.Conv2d(32, 64, 4, stride=2, padding=1), # 8x8
nn.ReLU(),
nn.Conv2d(64, 128, 4, stride=2, padding=1),# 4x4
nn.ReLU(),
)
self.fc_mu = nn.Linear(128 * 4 * 4, latent_dim)
self.fc_logvar = nn.Linear(128 * 4 * 4, latent_dim)
# 转置卷积解码器
self.decoder_fc = nn.Linear(latent_dim, 128 * 4 * 4)
self.decoder_conv = nn.Sequential(
nn.ConvTranspose2d(128, 64, 4, stride=2, padding=1), # 8x8
nn.ReLU(),
nn.ConvTranspose2d(64, 32, 4, stride=2, padding=1), # 16x16
nn.ReLU(),
nn.ConvTranspose2d(32, 3, 4, stride=2, padding=1), # 32x32
nn.Sigmoid()
)
def encode(self, x):
h = self.encoder_conv(x).view(x.size(0), -1)
return self.fc_mu(h), self.fc_logvar(h)
def reparameterize(self, mu, logvar):
std = torch.exp(0.5 * logvar)
eps = torch.randn_like(std)
return mu + eps * std
def decode(self, z):
h = F.relu(self.decoder_fc(z)).view(-1, 128, 4, 4)
return self.decoder_conv(h)
def forward(self, x):
mu, logvar = self.encode(x)
z = self.reparameterize(mu, logvar)
return self.decode(z), mu, logvar
4. 生成对抗网络(GAN)
GAN(Generative Adversarial Network)是 Ian Goodfellow 在 2014 年发表的论文(arXiv:1406.2661)中提出的革命性构想。
4.1 Generator 与 Discriminator 的博弈
GAN 由两个相互竞争的神经网络共同训练而成。
生成器(Generator, G)：从随机噪声 z 生成伪造数据。
- 目标：生成足以骗过 Discriminator 的逼真数据
判别器(Discriminator, D)：区分真实数据与生成的伪造数据。
- 目标：将真实数据判为 1，伪造数据判为 0
在这场博弈中，两个网络不断竞争学习，直至达到纳什均衡(Nash Equilibrium)。
4.2 极小极大损失(Minimax Loss)
min_G max_D [E_x[log D(x)] + E_z[log(1 - D(G(z)))]]
- D 最大化：在真实数据上最大化 log D(x)，在伪造数据上最大化 log(1 - D(G(z)))
- G 最小化：使 G(z)足以骗过 D，最小化 log(1 - D(G(z)))
实际训练中，G 的损失通常使用 -log(D(G(z)))(非饱和损失，non-saturating loss)。
4.3 基础 GAN 实现
import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import datasets, transforms
import numpy as np
class Generator(nn.Module):
"""GAN 生成器"""
def __init__(self, noise_dim=100, output_dim=784):
super().__init__()
self.model = nn.Sequential(
nn.Linear(noise_dim, 256),
nn.LeakyReLU(0.2),
nn.BatchNorm1d(256),
nn.Linear(256, 512),
nn.LeakyReLU(0.2),
nn.BatchNorm1d(512),
nn.Linear(512, 1024),
nn.LeakyReLU(0.2),
nn.BatchNorm1d(1024),
nn.Linear(1024, output_dim),
nn.Tanh() # 范围为 [-1, 1]
)
def forward(self, z):
return self.model(z).view(-1, 1, 28, 28)
class Discriminator(nn.Module):
"""GAN 判别器"""
def __init__(self, input_dim=784):
super().__init__()
self.model = nn.Sequential(
nn.Linear(input_dim, 1024),
nn.LeakyReLU(0.2),
nn.Dropout(0.3),
nn.Linear(1024, 512),
nn.LeakyReLU(0.2),
nn.Dropout(0.3),
nn.Linear(512, 256),
nn.LeakyReLU(0.2),
nn.Dropout(0.3),
nn.Linear(256, 1),
nn.Sigmoid()
)
def forward(self, x):
return self.model(x.view(x.size(0), -1))
def train_gan(epochs=200, noise_dim=100, lr=2e-4):
"""训练 GAN"""
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
transform = transforms.Compose([
transforms.ToTensor(),
transforms.Normalize([0.5], [0.5]) # 归一化到 [-1, 1] 范围
])
train_dataset = datasets.MNIST('./data', train=True, download=True, transform=transform)
dataloader = DataLoader(train_dataset, batch_size=64, shuffle=True)
G = Generator(noise_dim).to(device)
D = Discriminator().to(device)
# 分别使用独立的优化器
optimizer_G = optim.Adam(G.parameters(), lr=lr, betas=(0.5, 0.999))
optimizer_D = optim.Adam(D.parameters(), lr=lr, betas=(0.5, 0.999))
criterion = nn.BCELoss()
for epoch in range(epochs):
for i, (real_imgs, _) in enumerate(dataloader):
batch_size = real_imgs.size(0)
real_imgs = real_imgs.to(device)
real_labels = torch.ones(batch_size, 1).to(device)
fake_labels = torch.zeros(batch_size, 1).to(device)
# === 更新 Discriminator ===
optimizer_D.zero_grad()
# 真实图像损失
d_real = D(real_imgs)
d_loss_real = criterion(d_real, real_labels)
# 伪造图像损失
z = torch.randn(batch_size, noise_dim).to(device)
fake_imgs = G(z).detach() # 阻断 G 的梯度
d_fake = D(fake_imgs)
d_loss_fake = criterion(d_fake, fake_labels)
d_loss = d_loss_real + d_loss_fake
d_loss.backward()
optimizer_D.step()
# === 更新 Generator ===
optimizer_G.zero_grad()
z = torch.randn(batch_size, noise_dim).to(device)
fake_imgs = G(z)
g_pred = D(fake_imgs)
# G 试图骗过 D：让伪造样本被判为真实
g_loss = criterion(g_pred, real_labels)
g_loss.backward()
optimizer_G.step()
if (epoch + 1) % 20 == 0:
print(f"Epoch {epoch+1}/{epochs} | D Loss: {d_loss.item():.4f} | G Loss: {g_loss.item():.4f}")
return G, D
4.4 模式崩溃(Mode Collapse)问题
这是 GAN 最大的问题之一。Generator 不再生成多样化的图像，而是反复生成少数几种足以骗过 Discriminator 的模式，这一现象就叫模式崩溃。
5. GAN 发展史
5.1 DCGAN(Deep Convolutional GAN)
2015 年提出的 DCGAN，成功地将卷积神经网络应用到了 GAN 中。
class DCGANGenerator(nn.Module):
"""DCGAN 生成器（用于 64x64 图像）"""
def __init__(self, noise_dim=100, ngf=64, nc=3):
super().__init__()
self.main = nn.Sequential(
# 输入：noise_dim x 1 x 1
nn.ConvTranspose2d(noise_dim, ngf * 8, 4, 1, 0, bias=False),
nn.BatchNorm2d(ngf * 8),
nn.ReLU(True),
# 状态：(ngf*8) x 4 x 4
nn.ConvTranspose2d(ngf * 8, ngf * 4, 4, 2, 1, bias=False),
nn.BatchNorm2d(ngf * 4),
nn.ReLU(True),
# 状态：(ngf*4) x 8 x 8
nn.ConvTranspose2d(ngf * 4, ngf * 2, 4, 2, 1, bias=False),
nn.BatchNorm2d(ngf * 2),
nn.ReLU(True),
# 状态：(ngf*2) x 16 x 16
nn.ConvTranspose2d(ngf * 2, ngf, 4, 2, 1, bias=False),
nn.BatchNorm2d(ngf),
nn.ReLU(True),
# 状态：(ngf) x 32 x 32
nn.ConvTranspose2d(ngf, nc, 4, 2, 1, bias=False),
nn.Tanh()
# 输出：nc x 64 x 64
)
def forward(self, z):
z = z.view(z.size(0), -1, 1, 1)
return self.main(z)
class DCGANDiscriminator(nn.Module):
"""DCGAN 判别器（用于 64x64 图像）"""
def __init__(self, ndf=64, nc=3):
super().__init__()
self.main = nn.Sequential(
# 输入：nc x 64 x 64
nn.Conv2d(nc, ndf, 4, 2, 1, bias=False),
nn.LeakyReLU(0.2, inplace=True),
# 状态：(ndf) x 32 x 32
nn.Conv2d(ndf, ndf * 2, 4, 2, 1, bias=False),
nn.BatchNorm2d(ndf * 2),
nn.LeakyReLU(0.2, inplace=True),
# 状态：(ndf*2) x 16 x 16
nn.Conv2d(ndf * 2, ndf * 4, 4, 2, 1, bias=False),
nn.BatchNorm2d(ndf * 4),
nn.LeakyReLU(0.2, inplace=True),
# 状态：(ndf*4) x 8 x 8
nn.Conv2d(ndf * 4, ndf * 8, 4, 2, 1, bias=False),
nn.BatchNorm2d(ndf * 8),
nn.LeakyReLU(0.2, inplace=True),
# 状态：(ndf*8) x 4 x 4
nn.Conv2d(ndf * 8, 1, 4, 1, 0, bias=False),
nn.Sigmoid()
)
def forward(self, x):
return self.main(x).view(-1, 1)
5.2 WGAN(Wasserstein GAN)
WGAN(arXiv:1701.07875)使用 Wasserstein 距离取代 Jensen-Shannon 散度，大幅提升了训练的稳定性。
class WGANDiscriminator(nn.Module):
"""WGAN Critic（判别器）：不使用 Sigmoid"""
def __init__(self, input_dim=784):
super().__init__()
self.model = nn.Sequential(
nn.Linear(input_dim, 512),
nn.LeakyReLU(0.2),
nn.Linear(512, 256),
nn.LeakyReLU(0.2),
nn.Linear(256, 1)
# 不使用 Sigmoid！为了计算 Wasserstein 距离
)
def forward(self, x):
return self.model(x.view(x.size(0), -1))
def train_wgan(G, D, dataloader, device, epochs=100,
n_critic=5, clip_value=0.01, lr=5e-5):
"""训练 WGAN"""
optimizer_G = optim.RMSprop(G.parameters(), lr=lr)
optimizer_D = optim.RMSprop(D.parameters(), lr=lr)
for epoch in range(epochs):
for i, (real_imgs, _) in enumerate(dataloader):
real_imgs = real_imgs.to(device)
batch_size = real_imgs.size(0)
# 将 Critic 更新 n_critic 次
for _ in range(n_critic):
optimizer_D.zero_grad()
z = torch.randn(batch_size, 100).to(device)
fake_imgs = G(z).detach()
# Wasserstein 损失：E[D(real)] - E[D(fake)]
d_loss = -torch.mean(D(real_imgs)) + torch.mean(D(fake_imgs))
d_loss.backward()
optimizer_D.step()
# 权重裁剪（Lipschitz 条件）
for p in D.parameters():
p.data.clamp_(-clip_value, clip_value)
# 更新 Generator
optimizer_G.zero_grad()
z = torch.randn(batch_size, 100).to(device)
fake_imgs = G(z)
g_loss = -torch.mean(D(fake_imgs))
g_loss.backward()
optimizer_G.step()
5.3 WGAN-GP(Gradient Penalty)
WGAN 中的权重裁剪，被梯度惩罚(Gradient Penalty)取代，用来更好地强制满足 Lipschitz 条件。
def gradient_penalty(D, real_imgs, fake_imgs, device):
"""计算梯度惩罚"""
batch_size = real_imgs.size(0)
# 在真实图像与伪造图像之间随机插值
alpha = torch.rand(batch_size, 1, 1, 1).to(device)
interpolated = alpha * real_imgs + (1 - alpha) * fake_imgs
interpolated.requires_grad_(True)
d_interpolated = D(interpolated)
gradients = torch.autograd.grad(
outputs=d_interpolated,
inputs=interpolated,
grad_outputs=torch.ones_like(d_interpolated),
create_graph=True,
retain_graph=True
)[0]
gradients = gradients.view(batch_size, -1)
gradient_norm = gradients.norm(2, dim=1)
penalty = ((gradient_norm - 1) ** 2).mean()
return penalty
def wgan_gp_d_loss(D, real_imgs, fake_imgs, device, lambda_gp=10):
"""WGAN-GP 判别器损失"""
d_real = D(real_imgs).mean()
d_fake = D(fake_imgs).mean()
gp = gradient_penalty(D, real_imgs, fake_imgs, device)
return -d_real + d_fake + lambda_gp * gp
5.4 条件 GAN(cGAN)
通过添加标签条件，来生成特定类别的图像。
class ConditionalGenerator(nn.Module):
"""条件 GAN 生成器"""
def __init__(self, noise_dim=100, num_classes=10, embed_dim=50):
super().__init__()
self.label_embedding = nn.Embedding(num_classes, embed_dim)
self.model = nn.Sequential(
nn.Linear(noise_dim + embed_dim, 256),
nn.LeakyReLU(0.2),
nn.Linear(256, 512),
nn.LeakyReLU(0.2),
nn.Linear(512, 784),
nn.Tanh()
)
def forward(self, z, labels):
label_embed = self.label_embedding(labels)
x = torch.cat([z, label_embed], dim=1)
return self.model(x).view(-1, 1, 28, 28)
6. 扩散模型(Diffusion Models)
扩散模型是当前图像生成领域的标准做法，也是最新的生成模型。DDPM(Denoising Diffusion Probabilistic Models, Ho et al., 2020, arXiv:2006.11239)开创了这一领域。
6.1 扩散模型的直觉
扩散模型的核心思路在于两个过程。
Forward Process(添加噪声)：向真实图像逐步添加高斯噪声，最终变为纯噪声。经过 T 步之后，就会变成标准正态分布。
Reverse Process(去除噪声)：从纯噪声出发，逐步去除噪声，还原出真实图像。神经网络学习的正是这一逆向过程。
6.2 Forward Process 公式
每一步都会添加噪声。
q(x_t | x_{t-1}) = N(x_t; sqrt(1-β_t) * x_{t-1}, β_t * I)
这里的 β_t 就是噪声调度(noise schedule)。
一次性跳跃 t 步(重要特性)：
q(x_t | x_0) = N(x_t; sqrt(ā_t) * x_0, (1-ā_t) * I)
这里的 ā_t，是 s 从 1 到 t 变化时 (1 - β_s)的连乘积。
因此，可以直接计算出任意时间步 t 上的噪声图像。
import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np
class NoiseSchedule:
"""噪声调度管理"""
def __init__(self, timesteps=1000, beta_start=1e-4, beta_end=0.02):
self.timesteps = timesteps
# 线性噪声调度
self.betas = torch.linspace(beta_start, beta_end, timesteps)
self.alphas = 1.0 - self.betas
self.alphas_cumprod = torch.cumprod(self.alphas, dim=0)
self.alphas_cumprod_prev = F.pad(self.alphas_cumprod[:-1], (1, 0), value=1.0)
# Forward process 系数
self.sqrt_alphas_cumprod = torch.sqrt(self.alphas_cumprod)
self.sqrt_one_minus_alphas_cumprod = torch.sqrt(1.0 - self.alphas_cumprod)
# Reverse process 系数
self.posterior_variance = (
self.betas * (1.0 - self.alphas_cumprod_prev) /
(1.0 - self.alphas_cumprod)
)
def q_sample(self, x_start, t, noise=None):
"""Forward process：向 x_0 添加 t 步噪声"""
if noise is None:
noise = torch.randn_like(x_start)
# 提取对应 ā_t 的系数
sqrt_alphas_cumprod_t = self.sqrt_alphas_cumprod[t].view(-1, 1, 1, 1)
sqrt_one_minus_alphas_cumprod_t = self.sqrt_one_minus_alphas_cumprod[t].view(-1, 1, 1, 1)
return sqrt_alphas_cumprod_t * x_start + sqrt_one_minus_alphas_cumprod_t * noise
def predict_start_from_noise(self, x_t, t, noise):
"""从预测的噪声中还原 x_0"""
sqrt_recip_alphas_cumprod = 1.0 / self.sqrt_alphas_cumprod[t].view(-1, 1, 1, 1)
sqrt_recipm1_alphas_cumprod = (
torch.sqrt(1.0 / self.alphas_cumprod[t] - 1.0).view(-1, 1, 1, 1)
)
return sqrt_recip_alphas_cumprod * x_t - sqrt_recipm1_alphas_cumprod * noise
6.3 U-Net 架构
扩散模型的噪声预测网络使用 U-Net 结构，并通过正余弦嵌入对时间步 t 进行条件化。
class SinusoidalPositionEmbeddings(nn.Module):
"""用于时间步的正弦位置嵌入"""
def __init__(self, dim):
super().__init__()
self.dim = dim
def forward(self, time):
device = time.device
half_dim = self.dim // 2
embeddings = np.log(10000) / (half_dim - 1)
embeddings = torch.exp(torch.arange(half_dim, device=device) * -embeddings)
embeddings = time[:, None] * embeddings[None, :]
embeddings = torch.cat((embeddings.sin(), embeddings.cos()), dim=-1)
return embeddings
class ResidualBlock(nn.Module):
"""带有时间步条件化的残差块"""
def __init__(self, in_channels, out_channels, time_emb_dim):
super().__init__()
self.time_mlp = nn.Sequential(
nn.SiLU(),
nn.Linear(time_emb_dim, out_channels)
)
self.block1 = nn.Sequential(
nn.GroupNorm(8, in_channels),
nn.SiLU(),
nn.Conv2d(in_channels, out_channels, 3, padding=1)
)
self.block2 = nn.Sequential(
nn.GroupNorm(8, out_channels),
nn.SiLU(),
nn.Conv2d(out_channels, out_channels, 3, padding=1)
)
self.residual_conv = (
nn.Conv2d(in_channels, out_channels, 1)
if in_channels != out_channels else nn.Identity()
)
def forward(self, x, time_emb):
h = self.block1(x)
time_emb = self.time_mlp(time_emb)[:, :, None, None]
h = h + time_emb
h = self.block2(h)
return h + self.residual_conv(x)
class SimpleUNet(nn.Module):
"""简化版 U-Net（用于 DDPM）"""
def __init__(self, in_channels=1, model_channels=64, time_emb_dim=256):
super().__init__()
# 时间步嵌入
self.time_mlp = nn.Sequential(
SinusoidalPositionEmbeddings(model_channels),
nn.Linear(model_channels, time_emb_dim),
nn.SiLU(),
nn.Linear(time_emb_dim, time_emb_dim)
)
# 编码器
self.down1 = ResidualBlock(in_channels, model_channels, time_emb_dim)
self.down2 = ResidualBlock(model_channels, model_channels * 2, time_emb_dim)
self.pool = nn.MaxPool2d(2)
# 瓶颈层
self.bottleneck = ResidualBlock(model_channels * 2, model_channels * 2, time_emb_dim)
# 解码器
self.up1 = ResidualBlock(model_channels * 4, model_channels, time_emb_dim)
self.up2 = ResidualBlock(model_channels * 2, model_channels, time_emb_dim)
self.upsample = nn.Upsample(scale_factor=2, mode='bilinear', align_corners=True)
# 输出
self.final = nn.Conv2d(model_channels, in_channels, 1)
def forward(self, x, t):
time_emb = self.time_mlp(t)
# 编码器路径
x1 = self.down1(x, time_emb)
x2 = self.down2(self.pool(x1), time_emb)
# 瓶颈层
x_mid = self.bottleneck(x2, time_emb)
# 解码器路径（跳跃连接）
x = self.up1(torch.cat([self.upsample(x_mid), x2], dim=1), time_emb)
x = self.up2(torch.cat([self.upsample(x), x1], dim=1), time_emb)
return self.final(x)
6.4 DDPM 训练与采样
class DDPM:
"""DDPM 训练与采样"""
def __init__(self, model, noise_schedule, device):
self.model = model
self.schedule = noise_schedule
self.device = device
def get_loss(self, x_start, t):
"""训练损失：噪声预测误差"""
noise = torch.randn_like(x_start)
x_noisy = self.schedule.q_sample(x_start, t, noise)
# 神经网络预测所添加的噪声
predicted_noise = self.model(x_noisy, t)
# 简单的 MSE 损失
return F.mse_loss(noise, predicted_noise)
@torch.no_grad()
def p_sample(self, x_t, t):
"""Reverse process：从 x_t 一步去噪到 x_{t-1}"""
betas_t = self.schedule.betas[t].view(-1, 1, 1, 1).to(self.device)
sqrt_one_minus_alphas_cumprod_t = (
self.schedule.sqrt_one_minus_alphas_cumprod[t].view(-1, 1, 1, 1).to(self.device)
)
sqrt_recip_alphas_t = torch.sqrt(
1.0 / self.schedule.alphas[t]
).view(-1, 1, 1, 1).to(self.device)
# 预测的噪声
predicted_noise = self.model(x_t, t)
# 计算均值
model_mean = sqrt_recip_alphas_t * (
x_t - betas_t * predicted_noise / sqrt_one_minus_alphas_cumprod_t
)
if t[0] == 0:
return model_mean
else:
# 加上方差
posterior_variance_t = self.schedule.posterior_variance[t].view(-1, 1, 1, 1).to(self.device)
noise = torch.randn_like(x_t)
return model_mean + torch.sqrt(posterior_variance_t) * noise
@torch.no_grad()
def sample(self, batch_size, image_shape):
"""从纯噪声出发生成图像"""
img = torch.randn(batch_size, *image_shape).to(self.device)
for i in reversed(range(self.schedule.timesteps)):
t = torch.full((batch_size,), i, dtype=torch.long, device=self.device)
img = self.p_sample(img, t)
return img
def train_ddpm(epochs=100):
"""训练 DDPM"""
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
transform = transforms.Compose([
transforms.ToTensor(),
transforms.Normalize([0.5], [0.5]) # 归一化到 [-1, 1] 范围
])
dataset = datasets.MNIST('./data', train=True, download=True, transform=transform)
dataloader = DataLoader(dataset, batch_size=128, shuffle=True)
model = SimpleUNet(in_channels=1).to(device)
schedule = NoiseSchedule(timesteps=1000)
ddpm = DDPM(model, schedule, device)
optimizer = optim.Adam(model.parameters(), lr=2e-4)
for epoch in range(epochs):
total_loss = 0
for batch, (x, _) in enumerate(dataloader):
x = x.to(device)
# 随机采样时间步
t = torch.randint(0, schedule.timesteps, (x.size(0),), device=device)
loss = ddpm.get_loss(x, t)
optimizer.zero_grad()
loss.backward()
optimizer.step()
total_loss += loss.item()
if (epoch + 1) % 10 == 0:
avg_loss = total_loss / len(dataloader)
print(f"Epoch {epoch+1}/{epochs}, Loss: {avg_loss:.4f}")
# 生成样本
samples = ddpm.sample(16, (1, 28, 28))
print(f"Generated samples: {samples.shape}")
return model, ddpm
7. DDIM — 快速采样
DDPM 需要 1000 步采样，速度很慢。DDIM(Denoising Diffusion Implicit Models, Song et al., 2020)提供了更快的确定性采样方式。
7.1 DDIM 的思路
DDIM 将采样过程重新定义为一个非马尔可夫(non-Markovian)过程。核心在于沿用同一个训练好的模型，同时大幅减少采样步数(1000 → 50 步)。
@torch.no_grad()
def ddim_sample(model, schedule, batch_size, image_shape,
ddim_timesteps=50, eta=0.0, device='cpu'):
"""
DDIM 采样
eta=0.0：完全确定性
eta=1.0：等同于 DDPM
"""
# 以均匀间隔选取时间步
c = schedule.timesteps // ddim_timesteps
timestep_seq = list(range(0, schedule.timesteps, c))[::-1]
img = torch.randn(batch_size, *image_shape).to(device)
for i, t in enumerate(timestep_seq):
t_tensor = torch.full((batch_size,), t, dtype=torch.long, device=device)
t_prev = timestep_seq[i + 1] if i + 1 < len(timestep_seq) else -1
# 当前时间步的系数
alpha_bar = schedule.alphas_cumprod[t].to(device)
alpha_bar_prev = (
schedule.alphas_cumprod[t_prev].to(device) if t_prev >= 0
else torch.tensor(1.0, device=device)
)
# 预测噪声
pred_noise = model(img, t_tensor)
# 预测 x_0
pred_x0 = (img - torch.sqrt(1 - alpha_bar) * pred_noise) / torch.sqrt(alpha_bar)
pred_x0 = torch.clamp(pred_x0, -1, 1)
# 计算 sigma
sigma = eta * torch.sqrt(
(1 - alpha_bar_prev) / (1 - alpha_bar) * (1 - alpha_bar / alpha_bar_prev)
)
# 计算方向
pred_dir = torch.sqrt(1 - alpha_bar_prev - sigma**2) * pred_noise
# 下一步的图像
noise = torch.randn_like(img) if t_prev >= 0 else 0
img = torch.sqrt(alpha_bar_prev) * pred_x0 + pred_dir + sigma * noise
return img
8. Stable Diffusion 解析
Stable Diffusion(arXiv:2112.10752)是一种创新方法：不在像素空间，而是在潜在空间(Latent Space)中执行扩散过程。
8.1 潜在扩散模型(LDM)
若直接对高分辨率图像(512x512)应用扩散，计算成本会非常高。LDM 的做法是：
- 用 VAE 编码器将图像压缩到潜在空间(512x512x3 → 64x64x4)
- 在潜在空间中执行扩散
- 用 VAE 解码器还原出原始分辨率
由此，计算成本可以节省 8 倍以上。
8.2 Stable Diffusion 的组成部分
文本提示词 → CLIP 文本编码器 → 文本嵌入
↓
纯噪声 z_T → [U-Net with Cross-Attention] → 潜在向量 z_0
↓
VAE 解码器 → 最终图像
CLIP 文本编码器：将文本转换为有意义的向量表示。
U-Net with Cross-Attention：通过交叉注意力(cross-attention)，将文本嵌入用作条件。
Classifier-Free Guidance(CFG)：将无文本条件下的预测与有条件时的预测相结合，以提升文本还原度。
guided_noise = uncond_noise + guidance_scale * (cond_noise - uncond_noise)
8.3 使用 diffusers 库调用 Stable Diffusion
from diffusers import StableDiffusionPipeline, DPMSolverMultistepScheduler
import torch
# 加载模型
model_id = "stabilityai/stable-diffusion-2-1"
pipe = StableDiffusionPipeline.from_pretrained(
model_id,
torch_dtype=torch.float16,
use_safetensors=True,
)
# 使用更快的调度器
pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)
pipe = pipe.to("cuda")
# 生成图像
prompt = "a photorealistic landscape of mountains at sunset, 8k, highly detailed"
negative_prompt = "blurry, low quality, distorted"
image = pipe(
prompt,
negative_prompt=negative_prompt,
num_inference_steps=25, # 将 DDPM 的 1000 步缩减为 25 步
guidance_scale=7.5, # CFG 尺度
height=512,
width=512,
generator=torch.Generator("cuda").manual_seed(42)
).images[0]
image.save("generated_image.png")
# 图像到图像转换
from diffusers import StableDiffusionImg2ImgPipeline
from PIL import Image
img2img_pipe = StableDiffusionImg2ImgPipeline.from_pretrained(
model_id,
torch_dtype=torch.float16
).to("cuda")
init_image = Image.open("input.jpg").resize((512, 512))
result = img2img_pipe(
prompt="a painting in Van Gogh style",
image=init_image,
strength=0.75, # 保留原图的程度（0-1）
guidance_scale=7.5,
num_inference_steps=50
).images[0]
8.4 Inpainting(图像修复)
from diffusers import StableDiffusionInpaintPipeline
inpaint_pipe = StableDiffusionInpaintPipeline.from_pretrained(
"runwayml/stable-diffusion-inpainting",
torch_dtype=torch.float16
).to("cuda")
# 原始图像与蒙版
image = Image.open("photo.jpg").resize((512, 512))
mask = Image.open("mask.jpg").resize((512, 512)) # 白色 = 需要修复的区域
result = inpaint_pipe(
prompt="a beautiful garden with flowers",
image=image,
mask_image=mask,
num_inference_steps=50
).images[0]
9. ControlNet — 精细的图像控制
9.1 ControlNet 架构
ControlNet(Zhang et al., 2023)让 Stable Diffusion 能够以额外的控制信号(Canny 边缘、深度图、姿态等)作为条件。
它的做法是：固定原有的 U-Net 权重，另外添加一个独立的控制网络。在约 3.6 亿参数的 Stable Diffusion 之上，再额外训练一份约 3.6 亿参数的副本。
from diffusers import StableDiffusionControlNetPipeline, ControlNetModel
from diffusers.utils import load_image
import cv2
import numpy as np
# 加载 Canny ControlNet
controlnet = ControlNetModel.from_pretrained(
"lllyasviel/sd-controlnet-canny",
torch_dtype=torch.float16
)
pipe = StableDiffusionControlNetPipeline.from_pretrained(
"runwayml/stable-diffusion-v1-5",
controlnet=controlnet,
torch_dtype=torch.float16
).to("cuda")
# 提取 Canny 边缘
image = load_image("input.jpg")
image_array = np.array(image)
canny = cv2.Canny(image_array, 100, 200)
canny_image = Image.fromarray(canny)
# 使用 ControlNet 生成图像
result = pipe(
prompt="a beautiful oil painting",
image=canny_image, # 将 Canny 边缘用作控制信号
controlnet_conditioning_scale=1.0,
num_inference_steps=50,
guidance_scale=7.5
).images[0]
10. 最新的生成模型趋势
10.1 DiT(Diffusion Transformers)
Peebles 与 Xie(2022)提出的 DiT，用 Transformer 架构取代 U-Net，作为扩散模型的骨干网络。Sora、Flux、SD3 等最新模型都基于 DiT。
class DiTBlock(nn.Module):
"""Diffusion Transformer Block"""
def __init__(self, hidden_dim, num_heads, mlp_ratio=4.0):
super().__init__()
self.norm1 = nn.LayerNorm(hidden_dim)
self.attn = nn.MultiheadAttention(hidden_dim, num_heads, batch_first=True)
self.norm2 = nn.LayerNorm(hidden_dim)
mlp_dim = int(hidden_dim * mlp_ratio)
self.mlp = nn.Sequential(
nn.Linear(hidden_dim, mlp_dim),
nn.GELU(),
nn.Linear(mlp_dim, hidden_dim)
)
# AdaLN（Adaptive Layer Normalization）：时间步条件化
self.adaLN_modulation = nn.Sequential(
nn.SiLU(),
nn.Linear(hidden_dim, 6 * hidden_dim)
)
def forward(self, x, c):
# 从时间步/条件嵌入中计算调制参数
shift_msa, scale_msa, gate_msa, shift_mlp, scale_mlp, gate_mlp = (
self.adaLN_modulation(c).chunk(6, dim=-1)
)
# 调制后的归一化
x_norm = (1 + scale_msa.unsqueeze(1)) * self.norm1(x) + shift_msa.unsqueeze(1)
attn_out, _ = self.attn(x_norm, x_norm, x_norm)
x = x + gate_msa.unsqueeze(1) * attn_out
x_norm = (1 + scale_mlp.unsqueeze(1)) * self.norm2(x) + shift_mlp.unsqueeze(1)
x = x + gate_mlp.unsqueeze(1) * self.mlp(x_norm)
return x
10.2 SDXL(Stable Diffusion XL)
生成更高分辨率(1024x1024)图像的 Stable Diffusion 升级版本。
from diffusers import StableDiffusionXLPipeline
pipe = StableDiffusionXLPipeline.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0",
torch_dtype=torch.float16,
use_safetensors=True,
variant="fp16"
).to("cuda")
image = pipe(
prompt="a majestic lion in photorealistic style, 4k",
negative_prompt="cartoon, blurry",
height=1024,
width=1024,
num_inference_steps=50,
guidance_scale=5.0
).images[0]
10.3 生成模型比较
| 模型 | 年份 | 核心思路 | 优点 | 缺点 |
|---|---|---|---|---|
| VAE | 2013 | 学习潜在分布 | 稳定，潜在空间可解释 | 图像较模糊 |
| GAN | 2014 | 对抗训练 | 图像清晰锐利 | 训练不稳定，模式崩溃 |
| DDPM | 2020 | 迭代式去噪 | 高质量，多样性好 | 采样速度慢 |
| LDM | 2022 | 潜在空间扩散 | 高效，支持文本条件 | 结构较复杂 |
| DiT | 2022 | Transformer 骨干 | 扩展效率高 | 计算成本高 |
结语
我们完整地探索了生成式AI的旅程：从 VAE 的潜在空间理论开始，走过 GAN 的对抗博弈、DDPM 的迭代去噪，直至 Stable Diffusion 的潜在空间扩散。
这一领域发展得极快。今天的最新技术，明天就会成为基础。牢固地理解基础知识，并持续跟进最新论文，至关重要。
以下是几份推荐的持续学习资料。
- Hugging Face Diffusers：https://huggingface.co/docs/diffusers/
- VAE 原始论文：arXiv:1312.6114
- GAN 原始论文：arXiv:1406.2661
- DDPM 原始论文：arXiv:2006.11239
- LDM/Stable Diffusion：arXiv:2112.10752
参考资料
- Kingma, D. P., & Welling, M. (2013). Auto-Encoding Variational Bayes. arXiv:1312.6114
- Goodfellow, I., et al. (2014). Generative Adversarial Networks. arXiv:1406.2661
- Ho, J., et al. (2020). Denoising Diffusion Probabilistic Models. arXiv:2006.11239
- Rombach, R., et al. (2022). High-Resolution Image Synthesis with Latent Diffusion Models. arXiv:2112.10752
- Arjovsky, M., et al. (2017). Wasserstein GAN. arXiv:1701.07875
- Peebles, W., & Xie, S. (2022). Scalable Diffusion Models with Transformers. arXiv:2212.09748
- Hugging Face Diffusers: https://huggingface.co/docs/diffusers/
현재 단락 (1/805)
生成式AI(Generative AI)是当今技术领域最炙手可热的方向。DALL-E 3 仅凭一行文字就能生成照片般逼真的图像，Stable Diffusion 能创作艺术作品，Sora 则能生成视频...
작성 글자: 0자원문 글자: 27,323자작성 단락: 0/805