Watching Resilience4j Earn Its Keep: A Spring Boot Lab You Can Actually Poke At

Most Resilience4j write-ups read like a feature tour. You get an annotation table, a YAML block, and a line that says “now your service is resilient.” Then you ship it and never see the patterns fire — until two months later, an incident review forces you to find out whether the breaker ever opened. Spoiler: nobody’s sure.

This post takes the opposite approach. We’ll build a tiny Spring Boot application whose downstream dependency can be flipped between three behaviors at runtime — healthy, broken, and pathologically slow — and we’ll wire up enough logging that every protective layer announces itself in the console the moment it acts. By the end you can replay each scenario from a shell script and watch retries, the circuit transition, time-limiter cuts, bulkhead rejections, and rate-limiter denials happen in real time.

This isn’t theoretical. It’s about 200 lines of Java sitting in a working repo, and you can drive it from the terminal.

What We’re Building

One REST endpoint:

GET /api/payments/process

Behind it, five resilience patterns are stacked on a single service method. To make the patterns observable, we add two more endpoints that exist purely to break things on purpose:

POST /api/payments/mode/{SUCCESS|FAIL|SLOW} GET /api/payments/mode

The mode endpoint mutates the simulated downstream so we can drive it into each failure regime without restarting the JVM. Then a small Bash script (test-resilience.sh)

hammers /api/payments/process with each scenario in sequence.

The Fake Downstream

The whole demo hinges on a controllable dependency. Here’s the gist of PaymentClient:

public enum Mode { SUCCESS, FAIL, SLOW }

private final AtomicReference<Mode> mode = new AtomicReference<>(Mode.SUCCESS);

public String pay() {
      return switch (mode.get()) {
             case FAIL -> {
                throw new RuntimeException("Payment service unavailable");
             }
             case SLOW -> {
                Thread.sleep(5000);
                yield "Paid slowly txn-" + UUID.randomUUID();
            }
            case SUCCESS -> "Payment OK txn-" + UUID.randomUUID();
            };
 }

Three modes, one switch. FAIL throws immediately, which feeds errors into Retry and the Circuit Breaker. SLOW sleeps five seconds, which is longer than the time limiter allows and also long enough to clog the bulkhead.

The mode is stored in an AtomicReference so we can flip it via HTTP without restarting.

Stacking the Patterns

Here’s the only method that matters:

 @RateLimiter(name = "paymentRateLimiter")
 @Bulkhead(name = "paymentBulkhead")
 @Retry(name = "paymentRetry", fallbackMethod = "fallback")
 @CircuitBreaker(name = "paymentService")
 @TimeLimiter(name = "paymentTL")
 public CompletableFuture<String> processPayment() {
         return CompletableFuture.supplyAsync(() -> client.pay());
 }

 public CompletableFuture<String> fallback(Throwable t) {
         return CompletableFuture.completedFuture(
         "FALLBACK: payment temporarily unavailable, please retry shortly");
 }

Two things in this snippet matter more than they look:

The return type is CompletableFuture<String>, not String. This is non-negotiable. @TimeLimiter needs something it can cancel, and the easiest cancellable handle in the JDK is a CompletableFuture. Skip this detail and the time limiter silently does nothing.

The fallback lives on @Retry, not @CircuitBreaker. This is the bit that the docs glide over and that almost everyone gets wrong on the first try. Resilience4j applies the decorators inside-out: Bulkhead is innermost, then TimeLimiter, then RateLimiter, then CircuitBreaker, and finally Retry wraps everything. If you put fallbackMethod on @CircuitBreaker, the Circuit Breaker catches the exception and returns the fallback before Retry sees it. Retry receives a successful value, has nothing to retry, and disappears from your logs entirely. Move the fallback up to the outermost decorator (@Retry) and everything you’d expect to happen happens.

Configuration Without the Mystery

The YAML reads almost like the constraints in plain English. I’ll annotate each block with what it actually does:

resilience4j:
 circuitbreaker:
      instances:
       paymentService:
           slidingWindowType: COUNT_BASED
           slidingWindowSize: 10
           minimumNumberOfCalls: 5
           failureRateThreshold: 50
           waitDurationInOpenState: 10s
           permittedNumberOfCallsInHalfOpenState: 3
           automaticTransitionFromOpenToHalfOpenEnabled: true

Translation: track the last 10 calls, but don’t make decisions until at least 5 have happened. If the failure rate exceeds 50 percent, open the circuit and reject everything for 10 seconds, then let 3 probe requests through.

retry:
 instances:
  paymentRetry:
   maxAttempts: 3
   waitDuration: 500ms
   retryExceptions:
      - java.lang.RuntimeException
      - java.util.concurrent.TimeoutException

Three attempts, with half a second between them. The retryExceptions list is critical — Retry’s default behavior is “only retry the exceptions you explicitly list.” Skip this and your retries fire less often than you expect.

timelimiter:
  instances:
    paymentTL:
      timeoutDuration: 2s
      cancelRunningFuture: true

After two seconds, give up and cancel the future.

bulkhead:
 instances:
    paymentBulkhead:
       maxConcurrentCalls: 10
       maxWaitDuration: 100ms

At most 10 simultaneous calls into the protected method. The 11th waits 100 ms for a slot, then throws.

ratelimiter:
  instances:
     paymentRateLimiter:
        limitForPeriod: 5
        limitRefreshPeriod: 10s
        timeoutDuration: 0

Five permits, refilled every ten seconds. The timeoutDuration: 0 makes it strictly non-blocking: no permit, immediate rejection.

Making the Patterns Audible

The most useful piece of code in this whole demo is not the service method. It’s a small component that subscribes to every registry and turns each event into a slf4j log line:

@PostConstruct
void registerListeners() {

    circuitBreakerRegistry
            .circuitBreaker("paymentService")
            .getEventPublisher()
            .onStateTransition(event ->
                    log.info("[CB] State Transition: {}",
                            event.getStateTransition()))
            .onCallNotPermitted(event ->
                    log.warn("[CB] Call NOT permitted (Circuit Open)"))
            .onError(event ->
                    log.warn("[CB] Error recorded: {}",
                            event.getThrowable().toString()));

    retryRegistry
            .retry("paymentRetry")
            .getEventPublisher()
            .onRetry(event ->
                    log.warn("[RETRY] Attempt #{} after error: {}",
                            event.getNumberOfRetryAttempts(),
                            event.getLastThrowable()))
            .onError(event ->
                    log.error("[RETRY] Giving up after {} attempts",
                            event.getNumberOfRetryAttempts()));

    rateLimiterRegistry
            .rateLimiter("paymentRateLimiter")
            .getEventPublisher()
            .onFailure(event ->
                    log.warn("[RATE] Permit DENIED - Rate limit exceeded"));

    bulkheadRegistry
            .bulkhead("paymentBulkhead")
            .getEventPublisher()
            .onCallRejected(event ->
                    log.warn("[BULKHEAD] Call REJECTED - Bulkhead full"));

    timeLimiterRegistry
            .timeLimiter("paymentTL")
            .getEventPublisher()
            .onTimeout(event ->
                    log.warn("[TIMELIMITER] Call timed out"));
}

That’s the trick. Annotations alone make patterns exist. The event publishers make them visible. The first time I ran this with the listeners wired up, I caught a misconfigured retry within a minute — something I had previously stared at metrics dashboards for a week and missed.

Driving It From a Script

Curling endpoints manually is tedious. The test-resilience.sh script in the repo packs each scenario into a named command:

 ./test-resilience.sh fail # FAIL mode, fires 8 requests
 ./test-resilience.sh slow # SLOW mode, single request
 ./test-resilience.sh ratelimit # SUCCESS mode, fires 10 in a row
 ./test-resilience.sh bulkhead # SLOW mode, 25 in parallel
 ./test-resilience.sh full # everything in sequence with cooldowns

What each one provokes in the application log is the actually-interesting part.

The FAIL Scenario, Annotated

Switch the client to FAIL. Send one request. The console produces something like this:

[CB] error recorded: java.lang.RuntimeException: Payment service unavailable [RETRY] attempt #1 after error: Payment service unavailable [CB] error recorded: java.lang.RuntimeException: Payment service unavailable [RETRY] attempt #2 after error: Payment service unavailable [CB] error recorded: java.lang.RuntimeException: Payment service unavailable [RETRY] giving up after 3 attempts Fallback triggered for processPayment: RuntimeException — Payment service unavailable

That’s three exceptions, three retries — the Circuit Breaker records each one. Notice how Retry waited 500 ms between attempts, exactly what the YAML asked for.

Now keep firing. Around the fifth request — once the sliding window has 5 calls in it and 50 percent failure crosses the threshold — the breaker opens:

[CB] state transition: State transition from CLOSED to OPEN [CB] call NOT permitted (circuit open) [CB] call NOT permitted (circuit open)

From this moment on, the dependency isn’t even called. The fallback fires immediately. The downstream gets a break, your threads get a break, and your latency drops to single-digit milliseconds because nothing is actually happening except a rejection.

Ten seconds later (the waitDurationInOpenState):

[CB] state transition: State transition from OPEN to HALF_OPEN

Three probe requests go through. If they succeed, the breaker closes. If they fail, it opens again. This is the bit of the circuit breaker that’s hard to explain in prose and obvious in five seconds once you’ve watched it happen.

The SLOW Scenario

Flip to SLOW. The client sleeps 5 seconds, the time limiter is set to 2.

[TIMELIMITER] call timed out [CB] error recorded: java.util.concurrent.TimeoutException [RETRY] attempt #1 after error: java.util.concurrent.TimeoutException

Time limiter cancels the future at the 2-second mark. The cancellation surfaces as a TimeoutException. Because we listed TimeoutException in retryExceptions, Retry tries again. After three attempts the fallback fires. The user gets a response in roughly six seconds — slow, but bounded.

Without the time limiter, that same request would have taken at least 5 seconds per attempt, retries included, while occupying a thread the whole time.

The BULKHEAD Scenario

Flip to SLOW, then fire 25 requests in parallel. The bulkhead allows 10:

[BULKHEAD] call REJECTED — bulkhead full [BULKHEAD] call REJECTED — bulkhead full [BULKHEAD] call REJECTED — bulkhead full

15 of the 25 requests give up after the 100 ms maxWaitDuration and fall through to the fallback. The other 10 occupy slots while their slow calls finish.

This is the pattern that quietly saves you from cross-tenant interference. Without it, one slow downstream silently steals every thread your Tomcat connector has, and endpoints that have nothing to do with payments start returning 504s. With it, the damage is bounded to the budget you allocated.

The RATE Scenario

SUCCESS mode, ten calls back to back, five permits per 10 seconds:

[RATE] permit DENIED — rate limit exceeded ← request 6 [RATE] permit DENIED — rate limit exceeded ← request 7

Requests one through five succeed. The rest fail fast. This is the pattern people most often confuse with the bulkhead. The distinction is what they count: the rate limiter counts requests per unit time, the bulkhead counts requests currently running. A perfectly fast downstream can still trip the rate limiter; a perfectly steady downstream can still trip the bulkhead if a single call goes slow.

Things That Aren’t in the Tutorial

A handful of footnotes from building this:

Aspect order matters. Resilience4j applies decorators in a fixed order (Retry > CircuitBreaker > RateLimiter > TimeLimiter > Bulkhead, outermost to innermost). You can’t rearrange annotations to change it. If you need a different layering, you have to use the programmatic API.

fallbackMethod placement is load-bearing. As noted above, putting it on the wrong decorator can silently disable Retry. If you stack patterns, put the fallback on the outermost one you care about.

Retry has a closed exception list by default. Unlike most retry libraries, Resilience4j does not retry every Throwable by default. You declare retryExceptions explicitly. Forgetting this is a frequent source of “why didn’t it retry.”

TimeLimiter requires an asynchronous return type. A plain String return won’t be cancellable, so the time limiter has nothing to enforce. Wrap your call in CompletableFuture.supplyAsync.

Spring Boot 4 renamed spring-boot-starter-aop. If you’re on the bleeding edge like this project (Boot 4.1, Java 25), the AOP starter is now spring-boot-starter-aspectj. Without it, none of the annotations get woven in and nothing happens at runtime — silently.

What Resilience Actually Costs

Each pattern adds a small amount of overhead. The interesting cost isn’t CPU; it’s reasoning.

Once you stack five decorators, the failure modes multiply. A timeout can look like a retry storm. A bulkhead rejection can look like a circuit-open response. A misconfigured rate limit can mask a working downstream. Without the event publishers feeding into structured logs, you’re left guessing which layer fired.

The fix is the same fix every distributed-systems debugging exercise eventually arrives at: make every decision your application makes visible. Resilience4j gives you that visibility for free, but only if you wire the registries to a logger. That part is on you.

Closing Thought

Resilience isn’t about preventing failure — it’s about choosing how you’d rather fail, and then making that choice visible enough to debug.

GitHub-repo:https://github.com/RiteshGangthade/resilience4j-demo