0%
💡 왼쪽 원문을 읽으면서 오른쪽에 따라 써보세요. Tab 키로 힌트를 받을 수 있습니다.
自动驾驶 AI 完全指南
自动驾驶车辆是一套感知、预测、规划、控制这四个核心模块实时协同运作的复杂 AI 系统。本文将系统梳理从基于 LiDAR 的 3D 感知，到 BEVFormer 摄像头感知、多目标追踪、高精地图、端到端学习、仿真，直至功能安全的完整技术栈。
1. 自动驾驶技术栈概览
自动驾驶流水线大致由四个阶段构成。
- 感知（Perception）：从传感器数据中理解周边环境。融合 LiDAR、摄像头、雷达等多种传感器，进行 3D 目标检测并提取可行驶区域。
- 预测（Prediction）：预测周边车辆、行人的未来轨迹。基于 Transformer 的模型对复杂的交互关系进行建模。
- 规划（Planning）：生成通往目的地的最优路径。规则驱动方法与学习驱动方法并存。
- 控制（Control）：生成加速、制动、转向指令。借助 PID 控制器或模型预测控制（MPC）实现。
# 自动驾驶流水线抽象示例
class AutonomousDrivingStack:
def __init__(self, perception, prediction, planner, controller):
self.perception = perception
self.prediction = prediction
self.planner = planner
self.controller = controller
def step(self, sensor_data):
# 1. 感知
scene = self.perception.process(sensor_data)
# 2. 预测
future_states = self.prediction.forecast(scene)
# 3. 规划
trajectory = self.planner.plan(scene, future_states)
# 4. 控制
commands = self.controller.compute(trajectory)
return commands
2. LiDAR 点云 3D 感知
2.1 PointNet++
PointNet++ 是直接处理点云的分层神经网络。通过 Set Abstraction 层反复聚合局部结构。
import torch
import torch.nn as nn
class SetAbstraction(nn.Module):
"""PointNet++ Set Abstraction 层"""
def __init__(self, npoint, radius, nsample, in_channel, mlp):
super().__init__()
self.npoint = npoint
self.radius = radius
self.nsample = nsample
self.mlp_convs = nn.ModuleList()
self.mlp_bns = nn.ModuleList()
last_channel = in_channel
for out_channel in mlp:
self.mlp_convs.append(nn.Conv2d(last_channel, out_channel, 1))
self.mlp_bns.append(nn.BatchNorm2d(out_channel))
last_channel = out_channel
def forward(self, xyz, points):
# xyz: (B, N, 3), points: (B, C, N)
# 执行 farthest point sampling + ball query 后应用 MLP
new_xyz = self.farthest_point_sample(xyz, self.npoint)
grouped = self.ball_query(xyz, new_xyz, points)
for conv, bn in zip(self.mlp_convs, self.mlp_bns):
grouped = torch.relu(bn(conv(grouped)))
new_points = grouped.max(dim=-1)[0]
return new_xyz, new_points
def farthest_point_sample(self, xyz, npoint):
# FPS 实现（简化版）
B, N, _ = xyz.shape
centroids = torch.zeros(B, npoint, dtype=torch.long)
distance = torch.ones(B, N) * 1e10
farthest = torch.randint(0, N, (B,))
for i in range(npoint):
centroids[:, i] = farthest
centroid = xyz[torch.arange(B), farthest, :].unsqueeze(1)
dist = ((xyz - centroid) ** 2).sum(-1)
distance = torch.min(distance, dist)
farthest = distance.max(-1)[1]
return xyz[torch.arange(B).unsqueeze(1), centroids]
def ball_query(self, xyz, new_xyz, points):
# 半径内邻域点分组（简化版）
B, N, C = points.transpose(1, 2).shape
return points.unsqueeze(-1).expand(-1, -1, -1, self.nsample)
2.2 VoxelNet
VoxelNet 将点云划分为 3D 体素，通过 Voxel Feature Encoding（VFE）层提取特征，再送入 3D 卷积骨干网络。
class VoxelFeatureEncoder(nn.Module):
"""VoxelNet VFE 层"""
def __init__(self, in_channels=4, out_channels=128):
super().__init__()
self.fc = nn.Sequential(
nn.Linear(in_channels, 64),
nn.BatchNorm1d(64),
nn.ReLU(),
nn.Linear(64, out_channels),
nn.BatchNorm1d(out_channels),
nn.ReLU(),
)
def forward(self, voxel_features, num_points):
# voxel_features: (N_voxels, max_points, C)
B, M, C = voxel_features.shape
x = self.fc(voxel_features.view(-1, C))
x = x.view(B, M, -1)
# 仅对有效点掩码后做 max pooling
mask = torch.arange(M, device=x.device).unsqueeze(0) < num_points.unsqueeze(1)
x = x * mask.unsqueeze(-1).float()
return x.max(dim=1)[0]
3. BEVFormer：基于摄像头的 BEV 感知
BEVFormer 将多摄像头图像转换到 Bird's Eye View（BEV）空间，以完成 3D 目标检测。通过 Deformable Attention，将 3D 空间中的查询点投影到图像平面上并聚合特征。
import torch
import torch.nn as nn
class BEVFormerLayer(nn.Module):
"""BEVFormer Spatial Cross-Attention 层"""
def __init__(self, embed_dim=256, num_heads=8, num_cameras=6):
super().__init__()
self.num_cameras = num_cameras
self.embed_dim = embed_dim
self.attn = nn.MultiheadAttention(embed_dim, num_heads, batch_first=True)
self.norm1 = nn.LayerNorm(embed_dim)
self.norm2 = nn.LayerNorm(embed_dim)
self.ffn = nn.Sequential(
nn.Linear(embed_dim, embed_dim * 4),
nn.GELU(),
nn.Linear(embed_dim * 4, embed_dim),
)
def forward(self, bev_query, camera_features, lidar2img):
"""
bev_query: (B, H*W, C) - BEV 网格查询
camera_features: (B, N_cam, H_img, W_img, C) - 摄像头特征
lidar2img: (B, N_cam, 4, 4) - LiDAR 到图像的投影矩阵
"""
B, Q, C = bev_query.shape
# 将 BEV 查询转换为 3D 参考点，再投影到各摄像头
ref_3d = self.get_reference_points(B, device=bev_query.device)
projected = self.project_to_image(ref_3d, lidar2img) # (B, Q, N_cam, 2)
# 采样各摄像头特征后聚合
sampled = self.sample_camera_features(camera_features, projected)
# Cross-attention
attn_out, _ = self.attn(bev_query, sampled, sampled)
bev_query = self.norm1(bev_query + attn_out)
bev_query = self.norm2(bev_query + self.ffn(bev_query))
return bev_query
def get_reference_points(self, B, device, H=200, W=200, Z=8):
xs = torch.linspace(0, 1, W, device=device)
ys = torch.linspace(0, 1, H, device=device)
zs = torch.linspace(0, 1, Z, device=device)
grid = torch.stack(torch.meshgrid(xs, ys, zs), dim=-1)
return grid.view(1, -1, 3).expand(B, -1, -1)
def project_to_image(self, pts_3d, lidar2img):
# 将 3D 点投影为图像坐标（简化版）
return pts_3d[..., :2]
def sample_camera_features(self, features, coords):
# 用 grid_sample 采样特征（简化版）
B, Q, C = features.shape[0], coords.shape[1], features.shape[-1]
return torch.zeros(B, Q, C, device=features.device)
4. 3D 目标检测与多目标追踪
4.1 CenterPoint
CenterPoint 在 BEV 热力图中检测目标中心，并回归尺寸、朝向、速度。
class CenterPointHead(nn.Module):
"""CenterPoint 检测头"""
def __init__(self, in_channels=256, num_classes=10):
super().__init__()
self.heatmap = nn.Conv2d(in_channels, num_classes, 1)
self.offset = nn.Conv2d(in_channels, 2, 1) # x, y 偏移
self.height = nn.Conv2d(in_channels, 1, 1) # z
self.size = nn.Conv2d(in_channels, 3, 1) # w, l, h
self.rotation = nn.Conv2d(in_channels, 2, 1) # sin, cos
self.velocity = nn.Conv2d(in_channels, 2, 1) # vx, vy
def forward(self, x):
return {
'heatmap': self.heatmap(x).sigmoid(),
'offset': self.offset(x),
'height': self.height(x),
'size': self.size(x),
'rotation': self.rotation(x),
'velocity': self.velocity(x),
}
4.2 基于卡尔曼滤波器的多目标追踪
import numpy as np
class KalmanTracker:
"""用于 3D 目标追踪的卡尔曼滤波器"""
def __init__(self, initial_state):
# 状态向量：[x, y, z, vx, vy, vz, w, l, h, theta]
self.dt = 0.1
n = 10
self.x = np.array(initial_state, dtype=float).reshape(-1, 1)
self.P = np.eye(n) * 10.0 # 初始协方差
# 状态转移矩阵（匀速运动模型）
self.F = np.eye(n)
for i in range(3):
self.F[i, i + 3] = self.dt
# 观测矩阵（仅观测位置+尺寸+朝向）
self.H = np.zeros((7, n))
for i in range(7):
self.H[i, i] = 1.0
self.Q = np.eye(n) * 0.1 # 过程噪声
self.R = np.eye(7) * 1.0 # 观测噪声
def predict(self):
"""预测阶段"""
self.x = self.F @ self.x
self.P = self.F @ self.P @ self.F.T + self.Q
return self.x[:7].flatten()
def update(self, z):
"""更新阶段"""
z = np.array(z).reshape(-1, 1)
y = z - self.H @ self.x # 新息
S = self.H @ self.P @ self.H.T + self.R # 新息协方差
K = self.P @ self.H.T @ np.linalg.inv(S) # 卡尔曼增益
self.x = self.x + K @ y
self.P = (np.eye(len(self.x)) - K @ self.H) @ self.P
return self.x[:7].flatten()
class MOT3D:
"""基于匈牙利算法的 3D 多目标追踪"""
def __init__(self, max_age=3, min_hits=3, iou_threshold=0.3):
self.trackers = []
self.max_age = max_age
self.min_hits = min_hits
self.iou_threshold = iou_threshold
self.frame_count = 0
self.next_id = 0
def update(self, detections):
self.frame_count += 1
predictions = [t.predict() for t in self.trackers]
matched, unmatched_dets, unmatched_trks = self.associate(
detections, predictions
)
for d, t in matched:
self.trackers[t].update(detections[d])
for i in unmatched_dets:
trk = KalmanTracker(detections[i])
trk.id = self.next_id
self.next_id += 1
self.trackers.append(trk)
self.trackers = [
t for t in self.trackers
if t.time_since_update <= self.max_age
]
return self.trackers
def associate(self, detections, predictions):
# 基于 IoU 的匈牙利匹配（简化版）
if not predictions:
return [], list(range(len(detections))), []
return [], list(range(len(detections))), []
5. 高精地图与 SLAM
5.1 NDT 匹配
Normal Distribution Transform（NDT）将点云表示为正态分布，从而实现快速的扫描匹配。
class NDTMatching:
"""用于自动驾驶定位的 NDT 扫描匹配"""
def __init__(self, resolution=1.0):
self.resolution = resolution
self.voxel_map = {}
def build_map(self, points):
"""将点云转换为 NDT 体素地图"""
for pt in points:
key = tuple((pt / self.resolution).astype(int))
if key not in self.voxel_map:
self.voxel_map[key] = []
self.voxel_map[key].append(pt)
for key in self.voxel_map:
pts = np.array(self.voxel_map[key])
mean = pts.mean(axis=0)
cov = np.cov(pts.T) if len(pts) > 3 else np.eye(3)
self.voxel_map[key] = {'mean': mean, 'cov': cov}
def score(self, points, transform):
"""计算给定变换矩阵下的 NDT 分数"""
score = 0.0
R = transform[:3, :3]
t = transform[:3, 3]
for pt in points:
pt_transformed = R @ pt + t
key = tuple((pt_transformed / self.resolution).astype(int))
if key in self.voxel_map:
voxel = self.voxel_map[key]
diff = pt_transformed - voxel['mean']
score += np.exp(-0.5 * diff @ np.linalg.inv(voxel['cov']) @ diff)
return score
5.2 OpenDRIVE 解析
import xml.etree.ElementTree as ET
def parse_opendrive(xodr_path):
"""解析 OpenDRIVE 高精地图"""
tree = ET.parse(xodr_path)
root = tree.getroot()
roads = []
for road in root.findall('road'):
road_data = {
'id': road.get('id'),
'length': float(road.get('length')),
'lanes': [],
}
for lane_section in road.findall('.//laneSection'):
for lane in lane_section.findall('.//lane'):
lane_data = {
'id': int(lane.get('id')),
'type': lane.get('type'),
'speed': None,
}
speed = lane.find('.//speed')
if speed is not None:
lane_data['speed'] = float(speed.get('max', 0))
road_data['lanes'].append(lane_data)
roads.append(road_data)
return roads
6. 行为预测：基于 Transformer 的轨迹预测
class TrajectoryPredictor(nn.Module):
"""基于 Transformer 的多智能体轨迹预测"""
def __init__(self, input_dim=6, embed_dim=128, num_heads=4,
num_layers=3, pred_horizon=30, num_modes=6):
super().__init__()
self.input_proj = nn.Linear(input_dim, embed_dim)
encoder_layer = nn.TransformerEncoderLayer(
d_model=embed_dim, nhead=num_heads, batch_first=True
)
self.encoder = nn.TransformerEncoder(encoder_layer, num_layers=num_layers)
self.mode_query = nn.Embedding(num_modes, embed_dim)
decoder_layer = nn.TransformerDecoderLayer(
d_model=embed_dim, nhead=num_heads, batch_first=True
)
self.decoder = nn.TransformerDecoder(decoder_layer, num_layers=num_layers)
self.traj_head = nn.Linear(embed_dim, pred_horizon * 2)
self.prob_head = nn.Linear(embed_dim, 1)
self.num_modes = num_modes
def forward(self, agent_history, map_features=None):
"""
agent_history: (B, N_agents, T_hist, 6) - x,y,vx,vy,ax,ay
返回：trajectories (B, N_agents, K, T_pred, 2), probs (B, N_agents, K)
"""
B, N, T, C = agent_history.shape
x = self.input_proj(agent_history.view(B * N, T, C))
memory = self.encoder(x) # (B*N, T, embed_dim)
queries = self.mode_query.weight.unsqueeze(0).expand(B * N, -1, -1)
decoded = self.decoder(queries, memory) # (B*N, K, embed_dim)
trajs = self.traj_head(decoded) # (B*N, K, T_pred*2)
probs = self.prob_head(decoded).squeeze(-1).softmax(-1)
pred_horizon = trajs.shape[-1] // 2
trajs = trajs.view(B, N, self.num_modes, pred_horizon, 2)
probs = probs.view(B, N, self.num_modes)
return trajs, probs
7. 端到端自动驾驶
7.1 CARLA 仿真器集成
import carla
import numpy as np
def setup_carla_sensors(world, vehicle):
"""CARLA 传感器设置：摄像头 + LiDAR"""
bp_lib = world.get_blueprint_library()
# RGB 摄像头
cam_bp = bp_lib.find('sensor.camera.rgb')
cam_bp.set_attribute('image_size_x', '1280')
cam_bp.set_attribute('image_size_y', '720')
cam_bp.set_attribute('fov', '90')
cam_transform = carla.Transform(carla.Location(x=1.5, z=2.4))
camera = world.spawn_actor(cam_bp, cam_transform, attach_to=vehicle)
# LiDAR
lidar_bp = bp_lib.find('sensor.lidar.ray_cast')
lidar_bp.set_attribute('channels', '64')
lidar_bp.set_attribute('range', '100')
lidar_bp.set_attribute('points_per_second', '1200000')
lidar_bp.set_attribute('rotation_frequency', '20')
lidar_transform = carla.Transform(carla.Location(x=0.0, z=2.0))
lidar = world.spawn_actor(lidar_bp, lidar_transform, attach_to=vehicle)
return camera, lidar
def domain_randomization(world, weather_presets=None):
"""域随机化：降低 sim-to-real gap"""
import random
weather = carla.WeatherParameters(
cloudiness=random.uniform(0, 80),
precipitation=random.uniform(0, 30),
sun_altitude_angle=random.uniform(15, 90),
fog_density=random.uniform(0, 20),
wetness=random.uniform(0, 50),
)
world.set_weather(weather)
7.2 模仿学习（Behavior Cloning）
class BehaviorCloningAgent(nn.Module):
"""基于模仿学习的自动驾驶智能体"""
def __init__(self, img_channels=3, cmd_dim=4, action_dim=2):
super().__init__()
import torchvision.models as models
backbone = models.resnet34(pretrained=False)
self.visual_encoder = nn.Sequential(*list(backbone.children())[:-2])
self.pool = nn.AdaptiveAvgPool2d(1)
self.cmd_embed = nn.Embedding(cmd_dim, 64) # 指令：直行/左转/右转/掉头
self.policy = nn.Sequential(
nn.Linear(512 + 64, 256),
nn.ReLU(),
nn.Linear(256, 128),
nn.ReLU(),
nn.Linear(128, action_dim), # steer, throttle
nn.Tanh(),
)
def forward(self, image, command):
feat = self.pool(self.visual_encoder(image)).flatten(1)
cmd_feat = self.cmd_embed(command)
x = torch.cat([feat, cmd_feat], dim=-1)
return self.policy(x)
def dagger_data_augmentation(expert_data, agent_rollouts):
"""DAgger：通过数据聚合缓解 distributional shift"""
# 在智能体实际访问过的状态上收集专家标签
combined = {
'obs': torch.cat([expert_data['obs'], agent_rollouts['obs']]),
'actions': torch.cat([expert_data['actions'], agent_rollouts['expert_actions']]),
}
return combined
8. nuPlan 场景评估
from nuplan.planning.simulation.planner.abstract_planner import AbstractPlanner
class ReactiveIDMPlanner(AbstractPlanner):
"""兼容 nuPlan 的 IDM 规划器"""
def __init__(self, target_velocity=15.0, min_gap=2.0):
self.target_velocity = target_velocity
self.min_gap = min_gap
def initialize(self, initialization):
self.initialization = initialization
def name(self):
return 'ReactiveIDMPlanner'
def observation_type(self):
from nuplan.planning.simulation.observation.observation_type import DetectionsTracks
return DetectionsTracks
def compute_planner_trajectory(self, current_input):
ego_state = current_input.history.ego_states[-1]
observations = current_input.observations
# 用 IDM 模型计算纵向加速度
accel = self.idm_accel(ego_state, observations)
return self.build_trajectory(ego_state, accel)
def idm_accel(self, ego, obs):
v = ego.dynamic_car_state.speed
delta = 4.0 # 加速度指数
a_max = 2.0
s_star = self.min_gap + v * 1.5
return a_max * (1 - (v / self.target_velocity) ** delta
- (s_star / max(self.min_gap, 1.0)) ** 2)
def build_trajectory(self, ego_state, accel):
return []
9. 功能安全：ISO 26262 与 SOTIF
自动驾驶系统必须同时符合 ISO 26262（功能安全）和 SOTIF（ISO 21448，预期功能安全）。
- ASIL 分级：自动驾驶的核心功能要求达到 ASIL-D 等级，制动控制、转向控制均属此列。
- 硬件冗余：核心传感器与计算模块采用冗余设计，单点故障时切换至安全状态。
- 运行时监控：实时测量感知模型的不确定性（uncertainty），以检测系统性能下降。
- 边缘场景处理：除标准场景外，还需针对逆光、大雾、逆行车辆等定义 ODD（Operational Design Domain，设计运行域）边界，并执行安全停车流程。
class SafetyMonitor:
"""运行时安全监控器"""
def __init__(self, uncertainty_threshold=0.7, min_detection_confidence=0.5):
self.uncertainty_threshold = uncertainty_threshold
self.min_conf = min_detection_confidence
self.fallback_triggered = False
def check(self, perception_output):
uncertainty = perception_output.get('uncertainty', 0.0)
confidence = perception_output.get('confidence', 1.0)
if uncertainty > self.uncertainty_threshold:
self.trigger_fallback('high_uncertainty')
return False
if confidence < self.min_conf:
self.trigger_fallback('low_confidence')
return False
return True
def trigger_fallback(self, reason):
self.fallback_triggered = True
# MRC（Minimal Risk Condition）：执行安全停车
print(f'[Safety] Fallback triggered: {reason}')
测验
Q1. 为什么 Bird's Eye View（BEV）表示在自动驾驶感知中更受青睐？
答案：因为 BEV 能够在没有透视畸变的情况下，把 3D 空间表示为 2D 平面。
解析：在摄像头的透视图像中，物体大小会随距离变化，车道线或道路结构的几何关系也难以把握。转换为 BEV 后，所有物体都能以接近真实的尺寸和位置排列，可以直接用于路径规划。此外，将 LiDAR、雷达、摄像头等多种传感器的输出融合到同一个统一空间中也更加容易，与下游模块（预测、规划）之间的接口也随之简化。
Q2. 在传感器融合中，LiDAR 与摄像头之间互补的特性是什么？
答案：LiDAR 提供精确的 3D 距离信息，摄像头提供丰富的色彩/纹理信息。
解析：LiDAR 通过精密的点云测量距离与形状，但缺乏目标分类所需的外观信息。摄像头以高分辨率图像识别车牌、信号灯颜色、道路标志，但在深度估计上存在局限。将两种传感器结合后，可以用摄像头赋予语义标签，用 LiDAR 精确获取 3D 位置。在恶劣天气导致摄像头性能下降时，LiDAR 可以辅助；而在回反射表面导致 LiDAR 饱和时，摄像头则能加以弥补。
Q3. 卡尔曼滤波器在多目标追踪（MOT）中如何反复进行预测与更新？
答案：每一帧都用状态转移模型预测位置，再用新的检测结果修正误差。
解析：在预测阶段，利用前一状态（位置、速度）与状态转移矩阵（匀速运动模型）估计当前帧的位置，并传播协方差。在更新阶段，用卡尔曼增益对新检测结果（观测值）与预测值之差（新息）进行加权，从而修正状态。卡尔曼增益由预测不确定性与观测不确定性的比例决定——观测越准确，就越信任观测而非预测。在检测失败的帧中，只执行预测阶段以维持目标状态。
Q4. CARLA 仿真器中的域随机化（domain randomization）是如何缩小 sim-to-real gap 的？
答案：通过随机改变仿真参数，使模型在训练中学会对多样的视觉条件保持鲁棒。
解析：真实道路上的天气、光照、车辆颜色、路面反射率千差万别。在 CARLA 中随机改变天气（雨、雾、太阳高度角）、纹理、车流量、光照强度等条件进行训练，可以使模型不会对某个特定仿真环境的视觉偏差过拟合。这样一来，实际环境中会遇到的各种条件已经预先包含在训练分布之中，从而减少性能下降。此外，如果对摄像头内参、传感器噪声模型也做随机化，效果会更好。
Q5. 端到端自动驾驶中，模仿学习的 distributional shift 问题是什么，又该如何解决？
答案：一旦模型陷入训练中未见过的状态，误差就会不断累积。可以用 DAgger 或数据增强来缓解。
解析：行为克隆（Behavior Cloning）通过从专家轨迹中收集的（状态，行动）对进行学习。但在推理阶段，微小误差一旦累积，模型就会到达训练分布之外的状态，而由于没有学过该状态下的正确行动，误差便会像滚雪球一样越滚越大。DAgger（Dataset Aggregation）会在智能体实际访问到的状态上反复收集专家标签，从而扩充训练数据。此外，在仿真器中刻意生成恢复场景（如偏离车道后回归车道），或对观测添加噪声等数据增强方法，也同样有效。
参考资料
- BEVFormer: Learning Bird's Eye View Representation (Li et al., 2022)
- CenterPoint: Center-based 3D Object Detection (Yin et al., 2021)
- PointNet++: Deep Hierarchical Feature Learning (Qi et al., 2017)
- UniAD: Planning-oriented Autonomous Driving (Hu et al., 2023)
- nuPlan: A closed-loop ML-based planning benchmark (Caesar et al., 2021)
- ISO 26262: Road Vehicles – Functional Safety
- CARLA: An Open Urban Driving Simulator (Dosovitskiy et al., 2017)
현재 단락 (1/407)
自动驾驶车辆是一套感知、预测、规划、控制这四个核心模块实时协同运作的复杂 AI 系统。本文将系统梳理从基于 LiDAR 的 3D 感知，到 BEVFormer 摄像头感知、多目标追踪、高精地图、端到端...
작성 글자: 0자원문 글자: 15,110자작성 단락: 0/407