The Complete Autonomous Driving & Robotics Tech Stack: From C++, ROS2, CUDA, TensorRT to VLM/VLA, Simulation, and Beyond

┌────────────────────────────────────────────────────────────────┐
│             Autonomous Driving Tech Stack Architecture          │
│                                                                │
│  ┌──────────┐  ┌───────────┐  ┌───────────┐  ┌──────────────┐ │
│  │  Sensing  │  │ Perception│  │  Decision │  │   Control    │ │
│  │ GPS/IMU  │→│ CV/DL     │→│ Planning  │→│ Control      │ │
│  │ Camera   │  │ Sensor    │  │ Prediction│  │ CAN/Ethernet │ │
│  │ LiDAR    │  │ Fusion    │  │           │  │              │ │
│  │          │  │ VLM/VLA   │  │           │  │              │ │
│  └──────────┘  └───────────┘  └───────────┘  └──────────────┘ │
│                                                                │
│  ┌──────────────────────────────────────────────────────────┐  │
│  │ Infra Layer: C++ | ROS2 | CUDA | TensorRT | Cloud/MLOps │  │
│  └──────────────────────────────────────────────────────────┘  │
│                                                                │
│  ┌──────────────────────────────────────────────────────────┐  │
│  │ Validation Layer: SIL/HIL | Simulation(CARLA/Isaac) | VR/AR│
│  └──────────────────────────────────────────────────────────┘  │
└────────────────────────────────────────────────────────────────┘

// Concepts: Type-safe sensor interfaces
template<typename T>
concept Sensor = requires(T s) {
    { s.read() } -> std::convertible_to<SensorData>;
    { s.calibrate() } -> std::same_as<bool>;
};

// Ranges: Sensor data pipeline
auto obstacles = pointCloud
    | views::filter(isAboveGround)
    | views::transform(toWorldFrame)
    | views::take(maxObstacles);

// Coroutines: Cooperative multitasking without RTOS overhead
// Asynchronous I/O via co_await, co_yield

✗ Dynamic memory allocation on hot paths → ✓ std::pmr allocators or pre-allocated pools
✗ Exceptions in real-time control loops   → ✓ std::expected or error codes
✗ Mutex-based communication               → ✓ std::atomic, lock-free data structures
✗ Default scheduling                      → ✓ SCHED_FIFO / SCHED_RR (POSIX)

Front Camera ──┐
Left Camera  ──┤
Right Camera ──┼──→ [BEV Feature Space] ──→ 3D Detection
Rear Camera  ──┤                            Lane Detection
Side Cameras ──┘                            Occupancy Prediction

Perception Evolution:
CNN (2011-2016) → RNN+GAN (2016-2018) → BEV (2018-2020)
→ Transformer+BEV (2020-present) → Occupancy (2022-present) → End-to-End VLA (2024-present)

Action Output Comparison:

RT-2 Approach (Action Tokenization):
  "move arm" → LLM → [token256] [token128] [token064] → Discrete actions

pi0 Approach (Flow Matching):
  "move arm" → VLM → Flow Expert → Continuous vector field → Smooth actions

┌─────────────────────────────────────────────┐
│            CUDA Memory Hierarchy             │
│                                              │
│  Registers (per thread)                      │
│    ↓                                         │
│  Shared Memory (per block, ~48-164KB)        │
│    ↓                                         │
│  L2 Cache                                    │
│    ↓                                         │
│  Global Memory (VRAM)                        │
│                                              │
│  Thread → Warp(32) → Block(max 1024) → Grid  │
└─────────────────────────────────────────────┘

Before optimization: Conv → BatchNorm → ReLU (3 kernel launches)
After optimization:  Conv+BN+ReLU (1 kernel launch)

Impact: Up to 80% reduction in kernel launch overhead
        Up to 50% reduction in memory bandwidth
        ~30% throughput improvement

PyTorch Model
  → ONNX Export (torch.onnx.export)
  → TensorRT Builder (trtexec or Python API)
    → Graph optimization + layer fusion
    → Precision calibration (INT8/FP8)
    → Automatic kernel tuning
  → Serialized engine (.engine file)
  → TensorRT Runtime (inference)

Large Teacher (FP32)
  → Knowledge Distillation → Smaller Student
  → Structured Pruning → Remove channels/heads
  → QAT Fine-tuning → INT8/FP8
  → TensorRT Export → Fused and optimized engine

Predict:  x̂ₖ|ₖ₋₁ = F·x̂ₖ₋₁ + B·uₖ
          Pₖ|ₖ₋₁ = F·Pₖ₋₁·Fᵀ + Q

Update:   Kₖ = Pₖ|ₖ₋₁·Hᵀ·(H·Pₖ|ₖ₋₁·Hᵀ + R)⁻¹
          x̂ₖ = x̂ₖ|ₖ₋₁ + Kₖ·(zₖ - H·x̂ₖ|ₖ₋₁)

┌──────────────────────────────────────────────────┐
│                  SIL Environment                  │
│                                                   │
│  [Perception Algorithms] ←→ [Sensor Simulation]   │
│  [Planning Algorithms]   ←→ [Scenario Engine]     │
│  [Control Algorithms]    ←→ [Vehicle Dynamics Model]│
│                                                   │
│  Execution: Host PC (x86)                         │
│  Physical Hardware: None                          │
│  Iteration Speed: Seconds to minutes              │
│  CI/CD Integration: Yes (cloud parallelization)   │
└──────────────────────────────────────────────────┘

┌──────────────────────────────────────────────────┐
│                  HIL Environment                  │
│                                                   │
│  [Actual ECU (DUT)] ←→ [HIL Simulator]            │
│                         ├ Vehicle dynamics model   │
│                         ├ Sensor signal injection (HDMI/ETH)│
│                         ├ Bus simulation (CAN/ETH) │
│                         └ Fault injection          │
│                                                   │
│  Execution: Real target hardware (Orin, EyeQ, etc.)│
│  Real-time: Hardware clock rate                   │
│  ISO 26262: Required for functional safety certification│
└──────────────────────────────────────────────────┘

MIL (Model-in-the-Loop)          — MATLAB/Simulink prototyping
  → SIL                           — Host PC + simulation environment
    → PIL (Processor-in-the-Loop) — Target processor compilation, host execution
      → HIL                       — Target ECU + simulation environment
        → VIL (Vehicle-in-the-Loop) — Real vehicle + scenario injection
          → Road Testing          — Real vehicle + real environment

# Run CARLA via Docker
docker pull carlasim/carla:0.9.15
docker run --privileged --gpus all --net=host \
    carlasim/carla:0.9.15 /bin/bash ./CarlaUE4.sh

# Control scenarios via Python API
pip install carla

import carla

client = carla.Client('localhost', 2000)
world = client.get_world()

# Spawn a vehicle
blueprint = world.get_blueprint_library().find('vehicle.tesla.model3')
spawn_point = world.get_map().get_spawn_points()[0]
vehicle = world.spawn_actor(blueprint, spawn_point)

# Attach a camera sensor
camera_bp = world.get_blueprint_library().find('sensor.camera.rgb')
camera = world.spawn_actor(camera_bp, carla.Transform(), attach_to=vehicle)

┌─────────────────────────────────────────────────────────────────┐
│                   Full Autonomous Driving Stack                  │
│                                                                  │
│  1. Sensing         Sensor drivers, time sync, logging           │
│       ↓                                                          │
│  2. Localization    HD Map matching, V-SLAM, LiDAR SLAM, GNSS/IMU│
│       ↓              → 6-DOF vehicle pose (100+ Hz)              │
│  3. Perception      3D detection, tracking, semantic seg, Occupancy│
│       ↓              → 3D bounding boxes, track IDs, semantic map│
│  4. Prediction      Agent future trajectory prediction (3-8s)    │
│       ↓              → Multi-modal trajectories per agent        │
│  5. Planning        Route planning, behavior planning, motion planning│
│       ↓              → Trajectory (pose + velocity sequence)     │
│  6. Control         Lateral (steering) + longitudinal (accel/brake)│
│       ↓              → CAN commands (steer-by-wire, brake-by-wire)│
└─────────────────────────────────────────────────────────────────┘

Vehicle (Edge)
  → Upload raw logs via cellular/WiFi
  → Object Storage (S3/GCS/Azure Blob)
  → Data catalog & indexing (scenario mining)
  → Auto-annotation (pre-labeling with existing models)
  → Human annotation (verification, corner cases)
  → Dataset versioning (DVC, LakeFS)
  → Training cluster
  → Model registry
  → Validation pipeline (offline metrics, SIL)
  → OTA deployment

A/B Partitioning: Update inactive partition → switch on reboot
Delta Updates: Transmit only changed bytes (100-500MB vs 10+GB)
Staged Rollout: 1% → monitor → gradual expansion
Rollback: Revert to previous version on anomaly detection

Model deployment → Real-world driving data collection → Automatic failure case mining
→ Additional annotation → Retraining → SIL validation → A/B testing → Full deployment
→ [Repeat]