Chaos and Order

💡 왼쪽 원문을 읽으면서 오른쪽에 따라 써보세요. Tab 키로 힌트를 받을 수 있습니다.

Introduction
Understanding Cascading Failure
Circuit Breaker State Transitions
Resilience4j Implementation
Python Implementation (pybreaker)
Fallback Strategy Patterns
Combining Resilience Patterns
Prometheus Metrics
Testing
Quiz
Conclusion
References

Introduction

In a microservices architecture, inter-service calls are inevitable. However, when one service becomes slow or unresponsive, the calling service also suffers cascading failures. The Circuit Breaker pattern, like an electrical circuit breaker, cuts the circuit before failures propagate, protecting the stability of the entire system.

Understanding Cascading Failure

Normal state:
[Client] → [API Gateway] → [Order Service] → [Payment Service] → [Bank API]
                                              ↓
                                          [Inventory Service]

When Payment Service fails (without circuit breaker):
[Client] ← timeout ← [API Gateway] ← timeout ← [Order Service] ← timeout ← [Payment Service] ✗
                         Thread exhaustion!              Thread exhaustion!

Result: Payment failure brings down the entire system

Circuit Breaker State Transitions

        Failure rate less than threshold           Failure rate greater than or equal to threshold
    ┌────────────────────┐              ┌────────────────────┐
    │                    │              │                    │
    ▼                    │              │                    ▼
┌────────┐         ┌────────────┐         ┌──────────┐
│ CLOSED │ ──────> │ HALF-OPEN  │ <────── │   OPEN   │
│(Normal) │         │(Trial calls)│         │(Blocked)  │
└────────┘         └────────────┘         └──────────┘
    ▲                    │                     │
    │                    │                     │
    └────────────────────┘                     │
        Trial calls succeed                Wait duration elapsed
                                          (waitDurationInOpenState)

CLOSED: Normal state, all requests are allowed
OPEN: Failure detected, all requests fail immediately (fast fail)
HALF-OPEN: A limited number of trial calls are allowed; if successful, transitions back to CLOSED

Resilience4j Implementation

Adding Dependencies

<!-- Maven -->
<dependency>
    <groupId>io.github.resilience4j</groupId>
    <artifactId>resilience4j-spring-boot3</artifactId>
    <version>2.2.0</version>
</dependency>
<dependency>
    <groupId>io.github.resilience4j</groupId>
    <artifactId>resilience4j-circuitbreaker</artifactId>
</dependency>
<dependency>
    <groupId>io.github.resilience4j</groupId>
    <artifactId>resilience4j-retry</artifactId>
</dependency>
<dependency>
    <groupId>io.github.resilience4j</groupId>
    <artifactId>resilience4j-bulkhead</artifactId>
</dependency>

// Gradle
implementation 'io.github.resilience4j:resilience4j-spring-boot3:2.2.0'

Configuration (application.yml)

resilience4j:
  circuitbreaker:
    instances:
      paymentService:
        registerHealthIndicator: true
        slidingWindowType: COUNT_BASED
        slidingWindowSize: 10 # Based on last 10 requests
        minimumNumberOfCalls: 5 # Evaluate after at least 5 calls
        failureRateThreshold: 50 # 50%+ failure rate → OPEN
        slowCallRateThreshold: 80 # 80%+ slow calls → OPEN
        slowCallDurationThreshold: 2s # 2+ seconds → slow call
        waitDurationInOpenState: 30s # Wait before OPEN → HALF-OPEN
        permittedNumberOfCallsInHalfOpenState: 3 # Number of trial calls in HALF-OPEN
        automaticTransitionFromOpenToHalfOpenEnabled: true
        recordExceptions:
          - java.io.IOException
          - java.util.concurrent.TimeoutException
          - org.springframework.web.client.HttpServerErrorException
        ignoreExceptions:
          - com.example.BusinessException

  retry:
    instances:
      paymentService:
        maxAttempts: 3
        waitDuration: 1s
        enableExponentialBackoff: true
        exponentialBackoffMultiplier: 2
        retryExceptions:
          - java.io.IOException

  bulkhead:
    instances:
      paymentService:
        maxConcurrentCalls: 20
        maxWaitDuration: 500ms

  timelimiter:
    instances:
      paymentService:
        timeoutDuration: 3s
        cancelRunningFuture: true

Service Implementation

@Service
@Slf4j
public class PaymentService {

    private final RestTemplate restTemplate;

    @CircuitBreaker(name = "paymentService", fallbackMethod = "paymentFallback")
    @Retry(name = "paymentService")
    @Bulkhead(name = "paymentService")
    @TimeLimiter(name = "paymentService")
    public CompletableFuture<PaymentResponse> processPayment(PaymentRequest request) {
        log.info("Processing payment for order: {}", request.getOrderId());

        PaymentResponse response = restTemplate.postForObject(
            "http://payment-service/api/v1/payments",
            request,
            PaymentResponse.class
        );

        return CompletableFuture.completedFuture(response);
    }

    // Fallback method — called when circuit is OPEN
    private CompletableFuture<PaymentResponse> paymentFallback(
            PaymentRequest request, Exception ex) {
        log.warn("Payment circuit breaker activated for order: {}. Reason: {}",
                request.getOrderId(), ex.getMessage());

        // Strategy 1: Queue for later retry
        paymentRetryQueue.add(request);

        // Strategy 2: Return default response
        return CompletableFuture.completedFuture(
            PaymentResponse.builder()
                .orderId(request.getOrderId())
                .status(PaymentStatus.PENDING)
                .message("Payment is being processed. You will receive confirmation shortly.")
                .build()
        );
    }
}

Event Monitoring

@Component
public class CircuitBreakerEventListener {

    @Autowired
    private CircuitBreakerRegistry circuitBreakerRegistry;

    @PostConstruct
    public void registerEventListeners() {
        CircuitBreaker cb = circuitBreakerRegistry.circuitBreaker("paymentService");

        cb.getEventPublisher()
            .onStateTransition(event -> {
                log.warn("Circuit Breaker '{}' state transition: {} → {}",
                    event.getCircuitBreakerName(),
                    event.getStateTransition().getFromState(),
                    event.getStateTransition().getToState());

                // Slack/PagerDuty alerting
                if (event.getStateTransition().getToState() ==
                        CircuitBreaker.State.OPEN) {
                    alertService.sendAlert(
                        "CRITICAL: Payment circuit breaker OPEN!");
                }
            })
            .onError(event ->
                log.error("Circuit Breaker error: {} (duration: {}ms)",
                    event.getThrowable().getMessage(),
                    event.getElapsedDuration().toMillis())
            )
            .onSuccess(event ->
                log.debug("Circuit Breaker success (duration: {}ms)",
                    event.getElapsedDuration().toMillis())
            );
    }
}

Python Implementation (pybreaker)

import pybreaker
import requests
from functools import wraps

# Create circuit breaker
payment_breaker = pybreaker.CircuitBreaker(
    fail_max=5,              # OPEN after 5 failures
    reset_timeout=30,        # HALF-OPEN after 30 seconds
    exclude=[ValueError],    # Exclude business exceptions
)

# Register listener
class CircuitBreakerListener(pybreaker.CircuitBreakerListener):
    def state_change(self, cb, old_state, new_state):
        print(f"Circuit '{cb.name}': {old_state.name} → {new_state.name}")
        if new_state == pybreaker.STATE_OPEN:
            send_slack_alert(f"Warning: {cb.name} circuit OPEN!")

    def failure(self, cb, exc):
        print(f"Circuit '{cb.name}' failure: {exc}")

payment_breaker.add_listener(CircuitBreakerListener())


# Use as decorator
@payment_breaker
def process_payment(order_id: str, amount: float) -> dict:
    response = requests.post(
        "http://payment-service/api/v1/payments",
        json={"order_id": order_id, "amount": amount},
        timeout=3
    )
    response.raise_for_status()
    return response.json()


# Wrapper with fallback
def process_payment_safe(order_id: str, amount: float) -> dict:
    try:
        return process_payment(order_id, amount)
    except pybreaker.CircuitBreakerError:
        # Circuit OPEN state → immediate fallback
        return {
            "order_id": order_id,
            "status": "PENDING",
            "message": "Payment queued for retry"
        }
    except requests.RequestException as e:
        # Network error → failure recorded in circuit
        return {
            "order_id": order_id,
            "status": "FAILED",
            "error": str(e)
        }

Fallback Strategy Patterns

1. Cache Fallback

private PaymentResponse paymentCacheFallback(PaymentRequest req, Exception ex) {
    // Return last successful response from cache
    return cache.getIfPresent("payment:" + req.getOrderId());
}

2. Default Value

private List<Product> productFallback(String category, Exception ex) {
    // Return default recommended products
    return defaultProducts.getByCategory(category);
}

3. Alternative Service Call

private PaymentResponse paymentBackupFallback(PaymentRequest req, Exception ex) {
    // Call backup payment service
    return backupPaymentService.process(req);
}

4. Queueing for Async Processing

private PaymentResponse paymentQueueFallback(PaymentRequest req, Exception ex) {
    // Put in message queue for later processing
    kafkaTemplate.send("payment-retry", req);
    return PaymentResponse.pending(req.getOrderId());
}

Combining Resilience Patterns

// Application order (outer → inner):
// Retry → CircuitBreaker → RateLimiter → TimeLimiter → Bulkhead

@Retry(name = "service")           // 3. Retry on failure
@CircuitBreaker(name = "service")  // 2. Monitor failure rate
@Bulkhead(name = "service")        // 1. Limit concurrency
public Response callExternalService() {
    // ...
}

Request flow:
[Request] → Bulkhead (limit to 20 concurrent)
          → CircuitBreaker (monitor failure rate)
          → Retry (up to 3 retries on failure)
          → TimeLimiter (3-second timeout)
          → [External Service Call]

Prometheus Metrics

# application.yml
management:
  endpoints:
    web:
      exposure:
        include: health,metrics,prometheus
  metrics:
    distribution:
      percentiles-histogram:
        resilience4j.circuitbreaker.calls: true

# Circuit breaker state
resilience4j_circuitbreaker_state{name="paymentService"}
# 0=CLOSED, 1=OPEN, 2=HALF_OPEN

# Failure rate
resilience4j_circuitbreaker_failure_rate{name="paymentService"}

# Call statistics
rate(resilience4j_circuitbreaker_calls_total{name="paymentService",kind="successful"}[5m])
rate(resilience4j_circuitbreaker_calls_total{name="paymentService",kind="failed"}[5m])

# Slow call rate
resilience4j_circuitbreaker_slow_call_rate{name="paymentService"}

Testing

@SpringBootTest
class CircuitBreakerTest {

    @Autowired
    private CircuitBreakerRegistry registry;

    @Test
    void shouldOpenCircuitAfterFailures() {
        CircuitBreaker cb = registry.circuitBreaker("paymentService");

        // Verify CLOSED state
        assertThat(cb.getState()).isEqualTo(CircuitBreaker.State.CLOSED);

        // Simulate 5 failures
        for (int i = 0; i < 5; i++) {
            cb.onError(0, TimeUnit.MILLISECONDS, new IOException("timeout"));
        }

        // Verify transition to OPEN state
        assertThat(cb.getState()).isEqualTo(CircuitBreaker.State.OPEN);

        // Immediate failure in OPEN state
        assertThatThrownBy(() ->
            cb.decorateSupplier(() -> "test").get()
        ).isInstanceOf(CallNotPermittedException.class);
    }
}

Quiz

Q1. What are the three states of a circuit breaker?

CLOSED (normal, all calls allowed), OPEN (blocked, immediate failure), HALF-OPEN (limited trial calls allowed).

Q2. What does failureRateThreshold: 50 mean?

When the failure rate reaches 50% or above within the sliding window, the circuit transitions to the OPEN state.

Q3. What is the annotation application order in Resilience4j?

From outer to inner: Retry, then CircuitBreaker, then RateLimiter, then TimeLimiter, then Bulkhead.

Q4. What problem occurs in microservices without a circuit breaker?

Cascading Failure. A single slow service exhausts the threads of calling services, bringing down the entire system.

Q5. What happens when all trial calls in HALF-OPEN state succeed?

The circuit transitions back to the CLOSED state and resumes allowing all calls normally.

Q6. What is the role of the Bulkhead pattern?

It limits the number of concurrent calls, preventing a single service call from exhausting all threads. Like a ship's bulkhead, it isolates failures.

Q7. List the four fallback strategies.

Cache fallback (last successful response), default value return, alternative service call, and queueing for async processing.

Conclusion

The Circuit Breaker pattern is an essential fault isolation mechanism in microservices architecture. Resilience4j provides a lightweight and modular implementation, and when combined with other resilience patterns like Retry, Bulkhead, and TimeLimiter, it enables building robust distributed systems.