Skip to content
Published on

Istio Traffic Management Engine Deep Dive

🇰🇷KO🇺🇸EN🇯🇵JA
Split View
Authors

Introduction

Istio traffic management is a core domain, accounting for 40% of the exam scope. This post analyzes how Istio CRDs are translated into Envoy configurations and how Envoy actually processes traffic internally.

VirtualService to Envoy RouteConfiguration

Translation Pipeline

VirtualService (Istio CRD)
        │
        ▼ Pilot Translation Engine
Envoy Route Configuration
  ├── VirtualHost (host-based matching)
  │   ├── Route (path/header matching)
  │   │   ├── RouteAction (routing target)
  │   │   ├── WeightedCluster (weight-based splitting)
  │   │   └── RetryPolicy (retry policy)
  │   └── Route (default path)
  └── VirtualHost (other hosts)

Match Priority

VirtualService HTTP match rules are evaluated in definition order. The same order is maintained in Envoy RouteConfiguration:

  1. Most specific match first (exact > prefix > regex)
  2. If multiple match blocks exist, the first match applies
  3. Routes without match conditions act as catch-all

Traffic Shifting

Internal implementation of weight-based traffic splitting:

# Istio VirtualService
spec:
  http:
    - route:
        - destination:
            host: reviews
            subset: v1
          weight: 75
        - destination:
            host: reviews
            subset: v2
          weight: 25

This is translated to WeightedCluster in Envoy:

{
  "route": {
    "weighted_clusters": {
      "clusters": [
        {
          "name": "outbound|9080|v1|reviews.default.svc.cluster.local",
          "weight": 75
        },
        {
          "name": "outbound|9080|v2|reviews.default.svc.cluster.local",
          "weight": 25
        }
      ],
      "total_weight": 100
    }
  }
}

Envoy generates a random number for each request and selects a cluster based on weight ranges. This is probabilistic splitting, so it operates as an approximation rather than exactly 75:25.

DestinationRule to Envoy Cluster Configuration

Cluster Configuration Translation

DestinationRule is translated to Envoy Cluster configuration:

DestinationRule
  ├── host → Cluster name prefix
  ├── subsets → Individual Cluster creation
  │   ├── subset v1 → outbound|9080|v1|reviews.default.svc.cluster.local
  │   └── subset v2 → outbound|9080|v2|reviews.default.svc.cluster.local
  └── trafficPolicy → Cluster-level settings
      ├── connectionPool → circuit_breakers
      ├── outlierDetection → outlier_detection
      └── loadBalancer → lb_policy

Cluster Naming Convention

Envoy Cluster names follow this format:

direction|port|subset|FQDN

Examples:
outbound|9080|v1|reviews.default.svc.cluster.local
inbound|8080||productpage.default.svc.cluster.local

Load Balancing Algorithms

Round Robin (Default)

Request 1 → Endpoint A
Request 2 → Endpoint B
Request 3 → Endpoint C
Request 4 → Endpoint A  (cycles back)

Envoy implementation: Distributes requests to endpoints in order. With weights, operates as Weighted Round Robin.

Least Connections

Endpoint A: 3 active connections
Endpoint B: 1 active connection  ← next request goes here
Endpoint C: 5 active connections

Envoy implementation: O(1) operation to select the endpoint with the fewest active requests.

Random

Randomly selects an endpoint for each request
Statistically even distribution at scale

Consistent Hash

Used when session affinity is needed:

spec:
  trafficPolicy:
    loadBalancer:
      consistentHashLB:
        httpHeaderName: x-user-id

Envoy implementation: Based on the Ketama hash algorithm. Places endpoints on a hash ring and selects the nearest endpoint by hashing the request key.

Hash Ring:
   0 ─── EP_A ─── EP_B ─── EP_C ─── MAX
         │              │
    x-user-id:alice  x-user-id:bob
    (always EP_A)    (always EP_B)

Benefits:

  • Minimal remapping when endpoints are added/removed
  • Same key always goes to the same endpoint (sticky sessions)

Circuit Breaker Implementation

DestinationRule outlierDetection to Envoy Translation

# Istio DestinationRule
spec:
  trafficPolicy:
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 10s
      baseEjectionTime: 30s
      maxEjectionPercent: 50

Envoy behavior:

{
  "outlier_detection": {
    "consecutive_5xx": 5,
    "interval": "10s",
    "base_ejection_time": "30s",
    "max_ejection_percent": 50,
    "enforcing_consecutive_5xx": 100
  }
}

Circuit Breaker State Machine

[Closed] ──── consecutive 5xx >= 5 ────→ [Open/Ejected]
   ▲                                          │
   │                                   baseEjectionTime elapsed
   │                                          │
   └──────── successful request ──────── [Half-Open]
                                          (retry)

Detailed behavior:

  1. Closed: Normal state. All requests forwarded to the endpoint
  2. Open (Ejected): Consecutive error threshold exceeded. Endpoint removed from load balancing pool
  3. Half-Open: After baseEjectionTime, endpoint returns to pool. Ejection time increases on re-failure

Connection Pool Circuit Breaker

spec:
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100
      http:
        h2UpgradePolicy: DEFAULT
        http1MaxPendingRequests: 1024
        http2MaxRequests: 1024
        maxRequestsPerConnection: 10
        maxRetries: 3

Envoy internal behavior:

Request arrives
    │
    ▼
maxConnections (100) exceeded? ──→ 503 (overflow)
    │ No
    ▼
http1MaxPendingRequests (1024) exceeded? ──→ 503 (overflow)
    │ No
    ▼
http2MaxRequests (1024) exceeded? ──→ 503 (overflow)
    │ No
    ▼
Start processing request
    │
maxRequestsPerConnection (10) reached? ──→ Close connection, create new one

Retry Implementation

VirtualService retries to Envoy Translation

spec:
  http:
    - route:
        - destination:
            host: reviews
      retries:
        attempts: 3
        perTryTimeout: 2s
        retryOn: gateway-error,connect-failure,retriable-4xx

Envoy retry policy:

{
  "retry_policy": {
    "retry_on": "gateway-error,connect-failure,retriable-4xx",
    "num_retries": 3,
    "per_try_timeout": "2s",
    "retry_host_predicate": [
      {
        "name": "envoy.retry_host_predicates.previous_hosts"
      }
    ]
  }
}

Retry Behavior Detail

Original request → Endpoint A (fails, 503)
    │
Retry 1 → Endpoint B (fails, 503)  ← avoids A via previous_hosts
    │
Retry 2 → Endpoint C (fails, 503)  ← avoids A, B
    │
Retry 3 → Endpoint A (succeeds, 200)  ← reuses when candidates exhausted
    │
Final response: 200

Key points:

  • previous_hosts: Avoids previously failed hosts and retries on different endpoints
  • perTryTimeout: Per-attempt timeout (separate from overall timeout)
  • Retry budget: Envoy internally limits the number of concurrent retries

retryOn Conditions Detail

ConditionEnvoy Behavior
5xxWhen upstream returns a 5xx response
gateway-errorOn 502, 503, 504 responses
connect-failureOn TCP connection failure
retriable-4xxSpecific 4xx like 409 (Conflict)
refused-streamWhen upstream resets stream with REFUSED_STREAM
resetWhen connection is reset without a response

Fault Injection Implementation

Delay Injection

spec:
  http:
    - fault:
        delay:
          percentage:
            value: 10
          fixedDelay: 5s
      route:
        - destination:
            host: reviews

Envoy implementation: The envoy.filters.http.fault filter is inserted into the HTTP filter chain.

Request arrives
    │
    ▼
Fault Filter: Generate random number (0-100)
    │
    ├── random <= 10 → delay 5 seconds then pass to next filter
    │
    └── random > 10 → immediately pass to next filter
    │
    ▼
Router Filter → Forward to upstream

Abort Injection

spec:
  http:
    - fault:
        abort:
          percentage:
            value: 20
          httpStatus: 503
      route:
        - destination:
            host: reviews

Envoy implementation:

Request arrives
    │
    ▼
Fault Filter: Generate random number (0-100)
    │
    ├── random <= 20 → immediately return 503 (no upstream request)
    │
    └── random > 20 → pass to next filter

Combined Fault Injection

Delay and abort can be applied simultaneously:

spec:
  http:
    - fault:
        delay:
          percentage:
            value: 50
          fixedDelay: 3s
        abort:
          percentage:
            value: 10
          httpStatus: 500

Evaluation order: Delay is applied first, then abort is evaluated.

Traffic Mirroring Implementation

VirtualService mirror to Envoy Translation

spec:
  http:
    - route:
        - destination:
            host: reviews
            subset: v1
      mirror:
        host: reviews
        subset: v2
      mirrorPercentage:
        value: 100

Envoy implementation:

Request arrives
    │
    ├── Original request → reviews v1 (response returned to client)
    │
    └── Mirror request → reviews v2 (response ignored, fire-and-forget)
         │
         └── "-shadow" suffix added to Host header
             e.g., reviews → reviews-shadow

Key characteristics:

  • Mirror request responses are completely ignored
  • Mirror request failures do not affect the original request
  • "-shadow" is appended to the Host header to identify mirror traffic

ServiceEntry: Mesh Expansion

External Service Registration

apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: external-api
spec:
  hosts:
    - api.external-service.com
  location: MESH_EXTERNAL
  ports:
    - number: 443
      name: https
      protocol: TLS
  resolution: DNS

Resources created in Envoy:

ServiceEntry
    │
    ├── Listener: Filter chain added to 0.0.0.0:443 (outbound)
    │
    ├── Cluster: outbound|443||api.external-service.com
    │   ├── type: STRICT_DNS
    │   └── dns_lookup_family: V4_ONLY
    │
    └── Route: api.external-service.com → routes to that Cluster

Behavior by Resolution Mode

ResolutionEnvoy Cluster TypeBehavior
NONEORIGINAL_DSTForward to original request address
STATICSTATICUse IPs from endpoints field directly
DNSSTRICT_DNSPeriodic DNS resolution
DNS_ROUND_ROBINLOGICAL_DNSRound robin among DNS results

Advanced Routing Patterns

Header-Based Canary Deployment

spec:
  http:
    - match:
        - headers:
            x-canary:
              exact: 'true'
      route:
        - destination:
            host: reviews
            subset: v2
    - route:
        - destination:
            host: reviews
            subset: v1

URI-Based Routing

spec:
  http:
    - match:
        - uri:
            prefix: '/api/v2'
      route:
        - destination:
            host: api-v2
    - match:
        - uri:
            prefix: '/api'
      route:
        - destination:
            host: api-v1

Source-Based Routing

spec:
  http:
    - match:
        - sourceLabels:
            app: frontend
            version: v2
      route:
        - destination:
            host: reviews
            subset: v2
    - route:
        - destination:
            host: reviews
            subset: v1

Debugging Tools

istioctl Command Collection

# Check VirtualService application status
istioctl analyze -n NAMESPACE

# Check routing config for a specific service
istioctl proxy-config routes PODNAME -o json | python3 -m json.tool

# Check Envoy cluster statistics
istioctl proxy-config clusters PODNAME --fqdn reviews.default.svc.cluster.local

# Check circuit breaker status (Envoy admin port)
kubectl exec PODNAME -c istio-proxy -- curl -s localhost:15000/clusters | grep outlier

# Retry statistics
kubectl exec PODNAME -c istio-proxy -- curl -s localhost:15000/stats | grep retry

Common Problem Patterns

  1. 503 UC (Upstream Connection): Upstream connection failure - check connectionPool settings
  2. 503 UO (Upstream Overflow): Circuit breaker triggered - check maxConnections/maxPendingRequests
  3. 503 NR (No Route): Route matching failure - check VirtualService host/match
  4. 503 UH (Upstream Unhealthy): All endpoints ejected - check outlierDetection settings

Conclusion

Istio's traffic management engine is built on top of Envoy's powerful proxy capabilities. Istio CRDs are user-friendly abstractions, and actual behavior is determined by Envoy listener, route, cluster, and endpoint configurations.

When debugging traffic management issues, always verify the final configuration delivered to Envoy. Mastering the istioctl proxy-config commands enables rapid diagnosis of most traffic problems.

In the next post, we will examine the internals of the Istio security model.

Discuss on Twitter • View on GitHub