Split View: IPsec 완전 정복 — VPN 터널의 동작 원리부터 구축까지

IPsec (IP Security) = A security protocol suite operating at the IP layer (L3)

Components:
├── AH (Authentication Header): Authentication only (integrity + origin verification)
├── ESP (Encapsulating Security Payload): Encryption + Authentication (99% in practice)
├── IKE (Internet Key Exchange): Key exchange + SA negotiation
└── SA (Security Association): Security connection information store

SA = A "security agreement" between two devices

Included information:
├── SPI (Security Parameter Index): SA identifier (32-bit)
├── Encryption algorithm: AES-256-GCM
├── Authentication algorithm: SHA-256 HMAC
├── Key value: Shared secret key
├── Lifetime: 3600 seconds or 100GB
└── Mode: Tunnel / Transport

SAs are unidirectional → Bidirectional communication requires 2 SAs!

[Transport Mode]
  Host-to-host direct communication
  Original IP header preserved, only payload encrypted

  ┌──────────┬──────────┬────────────────┐
  │ IP Header│ESP Header│ Encrypted      │
  │ (original)│         │ Payload        │
  └──────────┴──────────┴────────────────┘

[Tunnel Mode] ← Site-to-Site VPN
  Gateway-to-gateway (wraps the entire original packet)
  New IP header added, original IP is also encrypted!

  ┌──────────┬──────────┬─────────────────────────┐
  │ New IP   │ESP Header│ Encrypted                │
  │ Header   │          │ [Original IP + Data]     │
  └──────────┴──────────┴─────────────────────────┘

ESP Packet Structure:

┌─────────────────────────────────────────────┐
│ IP Header (new, tunnel mode)                │
├─────────────────────────────────────────────┤
│ ESP Header                                  │
│  ├── SPI (32bit): Which SA to use           │
│  └── Sequence Number (32bit): Anti-replay   │
├─────────────────────────────────────────────┤
│ Encrypted Region                            │
│  ├── Original IP Header (tunnel mode)       │
│  ├── Original Payload (TCP/UDP + Data)      │
│  └── ESP Trailer (Padding + Next Header)    │
├─────────────────────────────────────────────┤
│ ESP Auth (ICV): HMAC-SHA256 (integrity)     │
└─────────────────────────────────────────────┘

# ESP encryption process (pseudocode)
from cryptography.hazmat.primitives.ciphers.aead import AESGCM
import os

def esp_encrypt(original_packet, sa):
    """ESP Encryption (AES-256-GCM)"""
    # 1. Get key and algorithm from SA
    key = sa.encryption_key  # 256-bit
    spi = sa.spi
    seq = sa.next_sequence_number()

    # 2. Create ESP Header
    esp_header = spi.to_bytes(4, 'big') + seq.to_bytes(4, 'big')

    # 3. Add Padding (align to block size)
    pad_len = (16 - (len(original_packet) + 2) % 16) % 16
    padding = bytes(range(1, pad_len + 1))
    esp_trailer = padding + bytes([pad_len, 4])  # 4 = IP-in-IP

    # 4. Encrypt (AES-256-GCM)
    nonce = os.urandom(12)
    aesgcm = AESGCM(key)
    plaintext = original_packet + esp_trailer
    ciphertext = aesgcm.encrypt(nonce, plaintext, esp_header)  # AAD = ESP Header

    # 5. New IP Header + ESP Header + Encrypted data
    new_ip = create_ip_header(sa.tunnel_src, sa.tunnel_dst)
    return new_ip + esp_header + nonce + ciphertext

Purpose: Establish a secure channel (to protect Phase 2)

[Initiator]                    [Responder]
    │                              │
    │──── SA Proposal ────────────▶│  Propose encryption/hash/DH group
    │◀─── SA Accepted ────────────│  Agreement
    │                              │
    │──── DH Public Value ────────▶│  Diffie-Hellman key exchange
    │◀─── DH Public Value ────────│  Both derive the same secret key!
    │                              │
    │──── Auth (PSK/Cert) ────────▶│  Pre-shared key or certificate
    │◀─── Auth ────────────────────│  Mutual authentication
    │                              │
    ╔══════════════════════════════╗
    ║  IKE SA established          ║
    ║  (encrypted channel)         ║
    ╚══════════════════════════════╝

Inside the channel protected by IKE SA:

[Initiator]                    [Responder]
    │                              │
    │──── IPsec SA Proposal ──────▶│  ESP/AH, encryption algorithm
    │◀─── IPsec SA Accepted ──────│  Agreement
    │                              │
    │──── Traffic Selector ────────▶│  Which IP/port ranges to protect?
    │◀─── Traffic Selector ────────│  Agreement
    │                              │
    ╔══════════════════════════════╗
    ║  IPsec SA established (x2,  ║
    ║  bidirectional)              ║
    ║  → Actual data encryption    ║
    ║    begins!                   ║
    ╚══════════════════════════════╝

# DH Key Exchange — Even an eavesdropper cannot learn the secret key!
p = 23  # Large prime (actual: 2048/4096 bit)
g = 5   # Generator

# Alice
a = 6   # Alice's secret value
A = pow(g, a, p)  # 5^6 mod 23 = 8  → Transmitted publicly

# Bob
b = 15  # Bob's secret value
B = pow(g, b, p)  # 5^15 mod 23 = 19 → Transmitted publicly

# Derive the shared secret key (same value!)
alice_secret = pow(B, a, p)  # 19^6 mod 23 = 2
bob_secret = pow(A, b, p)    # 8^15 mod 23 = 2

assert alice_secret == bob_secret  # ✅ Same secret key!
# Eavesdropper: Even knowing p=23, g=5, A=8, B=19,
# without knowing a or b, the secret key 2 cannot be computed
# (Discrete Logarithm Problem)

On-premises (192.168.1.0/24)
    │
    ├── Customer Gateway (IPsec)
    │       │
    │       │ ══ IPsec Tunnel 1 (Active) ══
    │       │ ══ IPsec Tunnel 2 (Standby) ══
    │       │
    ├── Virtual Private Gateway
    │
AWS VPC (10.0.0.0/16)

# IPsec VPN setup on Linux (strongSwan)
sudo apt install strongswan

# /etc/ipsec.conf
cat << 'EOF'
conn aws-vpn
    type=tunnel
    auto=start
    left=203.0.113.1          # Local public IP
    leftsubnet=192.168.1.0/24 # Local network
    right=52.10.20.30         # AWS VGW IP
    rightsubnet=10.0.0.0/16   # AWS VPC CIDR
    ike=aes256-sha256-modp2048
    esp=aes256-sha256
    keyexchange=ikev2
    authby=secret
EOF

# /etc/ipsec.secrets
echo '203.0.113.1 52.10.20.30 : PSK "MyPreSharedKey123"' | sudo tee /etc/ipsec.secrets

# Start
sudo ipsec restart
sudo ipsec status