✍️ 필사 모드: Kubernetes Networking Deep Dive Guide 2025: CNI, Service, Ingress, DNS, Network Policy

┌────────────────────────────────────────────────────┐
│       Kubernetes Networking Fundamentals           │
│                                                    │
│  ┌──────────┐           ┌──────────┐              │
│  │  Node 1  │           │  Node 2  │              │
│  │          │           │          │              │
│  │ ┌──────┐ │   No NAT  │ ┌──────┐ │              │
│  │ │Pod A │<├───────────>┤│Pod C │ │              │
│  │ │10.1.1│ │           │ │10.1.2│ │              │
│  │ └──────┘ │           │ └──────┘ │              │
│  │ ┌──────┐ │           │ ┌──────┐ │              │
│  │ │Pod B │ │           │ │Pod D │ │              │
│  │ │10.1.1│ │           │ │10.1.2│ │              │
│  │ └──────┘ │           │ └──────┘ │              │
│  └──────────┘           └──────────┘              │
│                                                    │
│  Pod A(10.1.1.2) -> Pod C(10.1.2.3): Direct comm  │
│  No NAT, no port mapping. Each Pod owns unique IP  │
└────────────────────────────────────────────────────┘

┌──────────────────────────────────────────┐
│       Kubernetes Networking Layers       │
├──────────────────────────────────────────┤
│ L7: Ingress / Gateway API               │
│     (HTTP routing, TLS termination)      │
├──────────────────────────────────────────┤
│ L4: Service                             │
│     (ClusterIP, NodePort, LoadBalancer)  │
├──────────────────────────────────────────┤
│ L3: Pod Networking (CNI)                │
│     (Pod-to-Pod, overlay/underlay)       │
├──────────────────────────────────────────┤
│ L2-L3: Node Networking                  │
│     (Physical/Virtual network)           │
└──────────────────────────────────────────┘

┌────────────────────────────────────────────┐
│                  Node                      │
│                                            │
│  ┌──────┐    veth pair    ┌──────────┐    │
│  │Pod A │<--------------->│           │    │
│  │eth0  │                │   cbr0    │    │
│  │      │                │  (bridge) │    │
│  └──────┘                │           │    │
│                          │           │    │
│  ┌──────┐    veth pair    │           │    │
│  │Pod B │<--------------->│           │    │
│  │eth0  │                └──────────┘    │
│  └──────┘                                │
│                                            │
│  1. Pod A sends packet to Pod B            │
│  2. Travels via veth pair to bridge (cbr0) │
│  3. Bridge finds destination veth by MAC   │
│  4. Delivered to Pod B via its veth pair    │
└────────────────────────────────────────────┘

┌──────────┐                      ┌──────────┐
│  Node 1  │                      │  Node 2  │
│          │                      │          │
│ ┌──────┐ │     VXLAN/IPinIP     │ ┌──────┐ │
│ │Pod A │ │    ┌───────────┐     │ │Pod C │ │
│ │10.1.1│ │--->│ Tunneling │---->│ │10.1.2│ │
│ └──────┘ │    │           │     │ └──────┘ │
│          │    │Encapsulate│     │          │
│ cbr0     │    │original   │     │ cbr0     │
│ 10.1.1.0 │    │in outer   │     │ 10.1.2.0 │
│          │    └───────────┘     │          │
│ eth0     │                      │ eth0     │
│192.168.1│                      │192.168.2│
└──────────┘                      └──────────┘

VXLAN Encapsulation:
┌─────────────────────────────────────────┐
│OuterIP │ UDP  │ VXLAN │InnerIP│ Payload │
│ Hdr    │ Hdr  │ Hdr   │ Hdr   │         │
│192->192│      │       │10->10 │ Data    │
└─────────────────────────────────────────┘

CNI Flow During Pod Creation:

1. kubelet requests container creation via CRI
2. Container runtime creates network namespace
3. kubelet invokes CNI plugin (ADD command)
4. CNI plugin:
   a. Creates veth pair
   b. Assigns IP address to Pod (IPAM)
   c. Configures routing rules
   d. Sets up overlay tunnel if needed
5. Pod is network-ready

# Calico BGP mode configuration (IPPool)
apiVersion: crd.projectcalico.org/v1
kind: IPPool
metadata:
  name: default-ipv4-ippool
spec:
  cidr: 10.244.0.0/16
  encapsulation: None        # No encapsulation with BGP
  natOutgoing: true
  nodeSelector: all()
  blockSize: 26              # /26 = 64 IPs per node

---
# Calico BGP Peering configuration
apiVersion: crd.projectcalico.org/v1
kind: BGPPeer
metadata:
  name: rack-tor-switch
spec:
  peerIP: 192.168.1.1
  asNumber: 64512
  nodeSelector: rack == "rack-1"

# Calico VXLAN mode (cloud environments)
apiVersion: crd.projectcalico.org/v1
kind: IPPool
metadata:
  name: default-ipv4-ippool
spec:
  cidr: 10.244.0.0/16
  encapsulation: VXLAN       # VXLAN encapsulation
  natOutgoing: true
  vxlanMode: Always

# Cilium Helm install (kube-proxy replacement mode)
# helm install cilium cilium/cilium --namespace kube-system \
#   --set kubeProxyReplacement=true \
#   --set k8sServiceHost=API_SERVER_IP \
#   --set k8sServicePort=6443 \
#   --set hubble.enabled=true \
#   --set hubble.relay.enabled=true \
#   --set hubble.ui.enabled=true \
#   --set encryption.enabled=true \
#   --set encryption.type=wireguard

# Verify Cilium status
# cilium status
# cilium connectivity test

# Cilium L7 Network Policy (HTTP-based policy)
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: l7-rule
  namespace: default
spec:
  endpointSelector:
    matchLabels:
      app: backend
  ingress:
  - fromEndpoints:
    - matchLabels:
        app: frontend
    toPorts:
    - ports:
      - port: "8080"
        protocol: TCP
      rules:
        http:
        - method: "GET"
          path: "/api/v1/.*"
        - method: "POST"
          path: "/api/v1/orders"

┌──────────────────────────────────────────────────┐
│               Service Traffic Flow                │
│                                                  │
│  Client Pod --> Service VIP --> kube-proxy        │
│                (10.96.0.10)    /iptables/IPVS    │
│                                    |             │
│                           ┌────────┼────────┐    │
│                           v        v        v    │
│                        Pod A    Pod B    Pod C    │
│                       10.1.1.2 10.1.1.3 10.1.2.4│
│                                                  │
│  Service VIP is not bound to any real interface   │
│  iptables/IPVS rules DNAT to Pod IPs             │
└──────────────────────────────────────────────────┘

# 1. ClusterIP (default) - internal access only
apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  type: ClusterIP
  selector:
    app: my-app
  ports:
  - port: 80
    targetPort: 8080
  # Cluster IP: auto-assigned (e.g., 10.96.0.10)
  # Accessible only within cluster at 10.96.0.10:80

# 2. NodePort - external access via fixed port on all nodes
apiVersion: v1
kind: Service
metadata:
  name: my-nodeport-service
spec:
  type: NodePort
  selector:
    app: my-app
  ports:
  - port: 80
    targetPort: 8080
    nodePort: 30080    # Range: 30000-32767
  # Accessible via port 30080 on any node
  # NodeIP:30080 -> ClusterIP:80 -> Pod:8080

# 3. LoadBalancer - auto-provisions cloud load balancer
apiVersion: v1
kind: Service
metadata:
  name: my-lb-service
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
spec:
  type: LoadBalancer
  selector:
    app: my-app
  ports:
  - port: 443
    targetPort: 8443
  # External LB IP -> NodePort -> ClusterIP -> Pod

# 4. ExternalName - maps external DNS to cluster service
apiVersion: v1
kind: Service
metadata:
  name: external-db
spec:
  type: ExternalName
  externalName: mydb.example.com
  # external-db.default.svc.cluster.local -> mydb.example.com

# 5. Headless Service - returns Pod IPs directly, no ClusterIP
apiVersion: v1
kind: Service
metadata:
  name: my-headless-service
spec:
  clusterIP: None            # Declares Headless
  selector:
    app: my-stateful-app
  ports:
  - port: 5432
  # DNS queries return individual Pod IPs, not Service IP
  # Used with StatefulSet: pod-0.my-headless-service.default.svc

apiVersion: v1
kind: Service
metadata:
  name: sticky-service
spec:
  selector:
    app: web
  ports:
  - port: 80
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 10800  # 3 hours
  # Same client IP routes to same Pod

iptables rule chain (per Service):

PREROUTING -> KUBE-SERVICES -> KUBE-SVC-XXXX -> KUBE-SEP-YYYY
                                               (probabilistic)

Service with 3 Pods:
KUBE-SVC-XXXX:
  -p tcp -d 10.96.0.10 --dport 80
    -> 33% probability KUBE-SEP-1 (DNAT -> 10.1.1.2:8080)
    -> 33% probability KUBE-SEP-2 (DNAT -> 10.1.1.3:8080)
    -> 34% probability KUBE-SEP-3 (DNAT -> 10.1.2.4:8080)

Problems:
- Rule count grows proportionally with Services/Pods O(n)
- Full chain rewrite on rule updates
- Severe degradation at 10,000+ Services

IPVS (IP Virtual Server):

┌───────────────────────────────────────────┐
│  IPVS Virtual Server: 10.96.0.10:80      │
│                                           │
│  Real Server 1: 10.1.1.2:8080  (weight 1)│
│  Real Server 2: 10.1.1.3:8080  (weight 1)│
│  Real Server 3: 10.1.2.4:8080  (weight 1)│
│                                           │
│  Algorithm: rr (Round Robin)              │
│  Others: lc, dh, sh, sed, nq             │
└───────────────────────────────────────────┘

Advantages:
- Hash table-based O(1) lookup
- Multiple load balancing algorithms
- Stable performance at scale
- Real-time statistics and connection tracking

# kube-proxy IPVS mode configuration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: "ipvs"
ipvs:
  scheduler: "rr"       # rr, lc, dh, sh, sed, nq
  syncPeriod: "30s"
  minSyncPeriod: "2s"

eBPF kube-proxy replacement:

┌────────────────────────────────────────────┐
│  Before: Pod -> iptables/IPVS -> Pod       │
│  eBPF:   Pod -> BPF program -> Pod (direct)│
│                                            │
│  ┌──────┐   BPF map    ┌──────┐           │
│  │Pod A │--(Service   --│Pod B │           │
│  │      │   lookup)     │      │           │
│  └──────┘               └──────┘           │
│                                            │
│  No iptables chain traversal!              │
│  Direct packet redirect in kernel space    │
└────────────────────────────────────────────┘

┌──────────────────────────────────────────────────┐
│                CoreDNS Operation                  │
│                                                  │
│  Pod --DNS query--> CoreDNS Pod                  │
│  (nameserver        (kube-system)                │
│   10.96.0.10)                                    │
│                      |                           │
│           ┌──────────┼──────────┐                │
│           v          v          v                │
│     K8s API     Corefile     Upstream            │
│   (Service/Pod  (Plugin      DNS                 │
│    records)      config)   (External DNS)        │
└──────────────────────────────────────────────────┘

Service DNS:
  my-service.my-namespace.svc.cluster.local
  |--svcname-| |-namespace--|

Pod DNS:
  10-1-1-2.my-namespace.pod.cluster.local
  |IP(dots->dashes)|

Headless Service Pod DNS (StatefulSet):
  pod-0.my-headless.my-namespace.svc.cluster.local
  |podname| |-svcname--|

SRV Records:
  _http._tcp.my-service.my-namespace.svc.cluster.local
  -> Returns port number and hostname

# CoreDNS Corefile (ConfigMap)
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health {
          lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
          ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf {
          max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }

# Create a debug Pod for DNS testing
kubectl run dnsutils --image=registry.k8s.io/e2e-test-images/jessie-dnsutils:1.3 \
  --restart=Never -- sleep infinity

# Service DNS lookup
kubectl exec dnsutils -- nslookup my-service.default.svc.cluster.local

# Pod DNS lookup
kubectl exec dnsutils -- nslookup 10-1-1-2.default.pod.cluster.local

# Detailed DNS response
kubectl exec dnsutils -- dig my-service.default.svc.cluster.local +search +short

# Check CoreDNS logs
kubectl logs -n kube-system -l k8s-app=kube-dns -f

# Check resolv.conf
kubectl exec dnsutils -- cat /etc/resolv.conf
# nameserver 10.96.0.10
# search default.svc.cluster.local svc.cluster.local cluster.local
# options ndots:5

# Problem with default ndots:5:
# Looking up "api.example.com" (fewer than 5 dots):
# 1. api.example.com.default.svc.cluster.local (fail)
# 2. api.example.com.svc.cluster.local (fail)
# 3. api.example.com.cluster.local (fail)
# 4. api.example.com (success)
# -> 3 unnecessary queries for external DNS lookups!

# Solution 1: Customize Pod DNS config
apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  dnsConfig:
    options:
    - name: ndots
      value: "2"       # Reduce ndots to 2
  containers:
  - name: app
    image: my-app

# Solution 2: Use FQDN directly (trailing dot)
# api.example.com.  <- trailing dot means FQDN

┌──────────────────────────────────────────────────┐
│               Ingress Architecture                │
│                                                  │
│  External Traffic                                │
│      |                                           │
│      v                                           │
│  ┌─────────────────────┐                         │
│  │  Ingress Controller │  (nginx, Traefik, etc.) │
│  │  (actual proxy)     │                         │
│  └─────────┬───────────┘                         │
│            |                                     │
│     Watches Ingress resources                    │
│     (routing rules)                              │
│            |                                     │
│     ┌──────┼──────┐                              │
│     v      v      v                              │
│   Svc A  Svc B  Svc C                            │
│   /api   /web   /docs                            │
└──────────────────────────────────────────────────┘

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: "50m"
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - app.example.com
    secretName: app-tls-secret
  rules:
  - host: app.example.com
    http:
      paths:
      - path: /api
        pathType: Prefix
        backend:
          service:
            name: api-service
            port:
              number: 80
      - path: /
        pathType: Prefix
        backend:
          service:
            name: frontend-service
            port:
              number: 80

┌──────────────────────────────────────────────────┐
│          Gateway API Resource Hierarchy           │
│                                                  │
│  Infrastructure Admin:                           │
│  ┌─────────────┐                                 │
│  │GatewayClass │  Which controller to use        │
│  └──────┬──────┘                                 │
│         |                                        │
│  Cluster Operator:                               │
│  ┌──────v──────┐                                 │
│  │  Gateway    │  Listeners (port, protocol, TLS)│
│  └──────┬──────┘                                 │
│         |                                        │
│  Application Developer:                          │
│  ┌──────v──────┐                                 │
│  │ HTTPRoute   │  Routing rules (host,path,hdr)  │
│  │ TCPRoute    │                                 │
│  │ GRPCRoute   │                                 │
│  └─────────────┘                                 │
└──────────────────────────────────────────────────┘

# 1. GatewayClass - Defined by infra admin
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: cilium
spec:
  controllerName: io.cilium/gateway-controller

---
# 2. Gateway - Defined by cluster operator
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: my-gateway
  namespace: gateway-infra
spec:
  gatewayClassName: cilium
  listeners:
  - name: https
    protocol: HTTPS
    port: 443
    tls:
      mode: Terminate
      certificateRefs:
      - kind: Secret
        name: wildcard-tls
    allowedRoutes:
      namespaces:
        from: Selector
        selector:
          matchLabels:
            shared-gateway: "true"

---
# 3. HTTPRoute - Defined by developer
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: app-routes
  namespace: my-app
spec:
  parentRefs:
  - name: my-gateway
    namespace: gateway-infra
  hostnames:
  - "app.example.com"
  rules:
  # Header-based routing
  - matches:
    - headers:
      - name: "X-Canary"
        value: "true"
    backendRefs:
    - name: app-canary
      port: 80
      weight: 100
  # Traffic splitting (Canary deployment)
  - matches:
    - path:
        type: PathPrefix
        value: /api
    backendRefs:
    - name: app-stable
      port: 80
      weight: 90
    - name: app-canary
      port: 80
      weight: 10
  # Default routing
  - backendRefs:
    - name: app-stable
      port: 80

# Deny all Ingress in namespace (default deny)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: production
spec:
  podSelector: {}           # Applies to all Pods
  policyTypes:
  - Ingress                 # Block Ingress only (Egress allowed)

---
# Deny all Egress too
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

# Backend Pod: Allow access only from frontend
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: backend-policy
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    # Allow only frontend Pods in same namespace
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 8080
  egress:
  # Allow DB access
  - to:
    - podSelector:
        matchLabels:
          app: postgres
    ports:
    - protocol: TCP
      port: 5432
  # Allow DNS (essential!)
  - to:
    - namespaceSelector: {}
      podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53

# Cilium: HTTP method/path based policy
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: api-l7-policy
spec:
  endpointSelector:
    matchLabels:
      app: api-server
  ingress:
  - fromEndpoints:
    - matchLabels:
        role: reader
    toPorts:
    - ports:
      - port: "443"
        protocol: TCP
      rules:
        http:
        - method: GET                # Read only
  - fromEndpoints:
    - matchLabels:
        role: writer
    toPorts:
    - ports:
      - port: "443"
        protocol: TCP
      rules:
        http:
        - method: GET
        - method: POST               # Write allowed too
        - method: PUT
        - method: DELETE

---
# Cilium: DNS-based Egress policy
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: dns-egress-policy
spec:
  endpointSelector:
    matchLabels:
      app: payment
  egress:
  - toFQDNs:
    - matchPattern: "*.stripe.com"   # Only Stripe API
    - matchName: "api.paypal.com"    # Only PayPal API
    toPorts:
    - ports:
      - port: "443"
        protocol: TCP

┌──────────────────────────────────────────────────┐
│              eBPF Architecture                    │
│                                                  │
│  User Space                                      │
│  ┌──────────────────────────────────────────┐   │
│  │  Cilium Agent / hubble                   │   │
│  │  (eBPF program mgmt, policy enforcement) │   │
│  └──────────────┬───────────────────────────┘   │
│                 | BPF Syscall                    │
│  ───────────────┼────────────────────────────── │
│  Kernel Space   |                                │
│  ┌──────────────v───────────────────────────┐   │
│  │  BPF Program (verified, JIT compiled)    │   │
│  │                                          │   │
│  │  Hook Points:                            │   │
│  │  ┌─────┐ ┌──────┐ ┌────────┐ ┌──────┐  │   │
│  │  │ XDP │ │tc/cls│ │ socket │ │kprobe│  │   │
│  │  └─────┘ └──────┘ └────────┘ └──────┘  │   │
│  │                                          │   │
│  │  BPF Maps (data sharing between progs)   │   │
│  │  ┌──────────────────────────────────┐   │   │
│  │  │ Service Map | Endpoint Map | CT  │   │   │
│  │  └──────────────────────────────────┘   │   │
│  └──────────────────────────────────────────┘   │
└──────────────────────────────────────────────────┘

# Monitor traffic with Hubble CLI
hubble observe --namespace production

# Check traffic for specific Pod
hubble observe --pod production/frontend-abc123

# Monitor DNS queries
hubble observe --protocol DNS

# Check dropped traffic (Network Policy)
hubble observe --verdict DROPPED

# Monitor HTTP requests (L7)
hubble observe --protocol HTTP --http-method GET

# Example output:
# TIMESTAMP            SOURCE              DESTINATION         TYPE     VERDICT
# Apr 14 09:15:23.456  prod/frontend-xxx   prod/backend-yyy    L4/TCP   FORWARDED
# Apr 14 09:15:23.789  prod/backend-yyy    prod/postgres-zzz   L4/TCP   FORWARDED
# Apr 14 09:15:24.012  prod/frontend-xxx   prod/redis-aaa      L4/TCP   DROPPED

# Run tcpdump via ephemeral container (K8s 1.25+)
kubectl debug -it pod/my-app-xxx \
  --image=nicolaka/netshoot \
  --target=my-app-container \
  -- tcpdump -i eth0 -nn port 8080

# Capture traffic with specific host
kubectl debug -it pod/my-app-xxx \
  --image=nicolaka/netshoot \
  --target=my-app-container \
  -- tcpdump -i eth0 -nn host 10.1.2.3

# Capture DNS queries
kubectl debug -it pod/my-app-xxx \
  --image=nicolaka/netshoot \
  --target=my-app-container \
  -- tcpdump -i eth0 -nn port 53

# Comprehensive testing with netshoot Pod
kubectl run netshoot --image=nicolaka/netshoot --rm -it -- bash

# TCP connection test
curl -v telnet://my-service:8080

# DNS resolution check
dig my-service.default.svc.cluster.local
dig +trace my-service.default.svc.cluster.local

# MTU check
ping -M do -s 1472 target-pod-ip  # 1472 + 28 = 1500

# Routing table check
ip route show

# ARP table check
ip neigh show

# iptables rules check (on node)
iptables -t nat -L KUBE-SERVICES -n --line-numbers

# 1. Check current Network Policies
kubectl get networkpolicy -A

# 2. Check policies applied to specific Pod
kubectl describe networkpolicy -n production

# 3. For Cilium: endpoint policy status
cilium endpoint list
cilium policy get

# 4. Connection test
kubectl exec -it frontend-pod -- curl -v http://backend-service:8080

# 5. Verify DNS is allowed (essential for Egress Policy)
kubectl exec -it my-pod -- nslookup kubernetes.default

# Common mistakes:
# - Missing DNS (UDP/TCP 53) in Egress Policy
# - Using podSelector without namespaceSelector
#   (won't match Pods in other namespaces)
# - Missing Egress in policyTypes (Egress rules ignored)

# 1. Check CoreDNS Pod status
kubectl get pods -n kube-system -l k8s-app=kube-dns

# 2. Check CoreDNS logs
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=100

# 3. Check DNS service endpoints
kubectl get endpoints kube-dns -n kube-system

# 4. Check resolv.conf
kubectl exec my-pod -- cat /etc/resolv.conf

# 5. Direct DNS query test
kubectl exec my-pod -- nslookup kubernetes.default.svc.cluster.local 10.96.0.10

# 6. External DNS resolution test
kubectl exec my-pod -- nslookup google.com

# For DNS performance issues:
# - Reduce ndots:5 to ndots:2
# - Use NodeLocal DNSCache
# - Enable autopath plugin

MTU Chain:
Pod (MTU) -> Overlay (VXLAN -50, IPinIP -20) -> Physical NIC (MTU)

Recommended settings:
- Physical NIC: Jumbo Frame 9000 (if possible)
- VXLAN overlay: Physical MTU - 50 = 8950
- IPinIP overlay: Physical MTU - 20 = 8980
- WireGuard encryption: Physical MTU - 60 = 8940

Default 1500 NIC:
- VXLAN: 1500 - 50 = 1450
- IPinIP: 1500 - 20 = 1480

# Calico MTU configuration
apiVersion: crd.projectcalico.org/v1
kind: FelixConfiguration
metadata:
  name: default
spec:
  mtu: 8950              # Physical NIC 9000 - VXLAN 50
  vxlanMTU: 8950
  wireguardMTU: 8940

# NodeLocal DNSCache DaemonSet (simplified)
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-local-dns
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: node-local-dns
  template:
    spec:
      containers:
      - name: node-cache
        image: registry.k8s.io/dns/k8s-dns-node-cache:1.23.0
        args:
        - "-localip"
        - "169.254.20.10"    # Link-local IP
        - "-conf"
        - "/etc/Corefile"
        - "-upstreamsvc"
        - "kube-dns"

NodeLocal DNSCache Flow:

Pod -> 169.254.20.10 (NodeLocal) -> CoreDNS (on cache miss)
                |
          On cache hit: immediate response

Benefits:
- DNS latency reduced by 50%+
- CoreDNS load reduced by 70-80%
- Eliminates conntrack contention
- Prevents UDP DNS packet loss

# Node-level kernel parameter tuning

# Connection tracking table size
net.netfilter.nf_conntrack_max = 1048576

# TCP buffer sizes
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

# TCP connection reuse
net.ipv4.tcp_tw_reuse = 1

# SYN backlog
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535

# Keep-alive
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_keepalive_probes = 10

┌──────────────┐         ┌──────────────┐
│  Cluster A   │         │  Cluster B   │
│  (us-east-1) │         │  (eu-west-1) │
│              │         │              │
│  ┌────────┐  │ Tunnel  │  ┌────────┐  │
│  │Cilium  │<-┼---------┼->│Cilium  │  │
│  │Agent   │  │         │  │Agent   │  │
│  └────────┘  │         │  └────────┘  │
│              │         │              │
│  Pod CIDR:   │         │  Pod CIDR:   │
│  10.1.0.0/16 │         │  10.2.0.0/16 │
│              │         │              │
│  Service:    │         │  Service:    │
│  shared-db   │ Global  │  shared-db   │
│  (annotated) │ Service │  (annotated) │
└──────────────┘         └──────────────┘

# Global Service configuration (on both clusters)
apiVersion: v1
kind: Service
metadata:
  name: shared-database
  annotations:
    service.cilium.io/global: "true"
    service.cilium.io/shared: "true"
spec:
  selector:
    app: postgres
  ports:
  - port: 5432