One Slow Partner API Took Down Four Unrelated Systems — Fixing It with the Bulkhead Pattern and Apache Flink

The Problem

Large integration platforms rarely talk to just one system. A single event — a transaction, a claim, a customer update — usually fans out to several places at once: fraud scoring, notifications, audit logging, a partner API, an analytics pipeline. Each of these is a separate concern. They shouldn’t depend on each other.

But in practice, they often share the same underlying resources. The same thread pool. The same database connection pool. The same Kafka consumer group. And that shared foundation quietly turns four independent integrations into one fragile chain.

Here’s what that looks like in code — a common but dangerous pattern:

@Service
public class IntegrationDispatcher {

    // one shared executor for everything
    private final ExecutorService executor = Executors.newFixedThreadPool(20);

    public void dispatch(TransactionEvent event) {
        executor.submit(() -> fraudService.score(event));
        executor.submit(() -> notificationService.notify(event));
        executor.submit(() -> auditService.log(event));
        executor.submit(() -> partnerBankClient.sync(event)); // slow, external, unreliable
    }
}

One day, the partner bank’s API starts responding slowly — 8 seconds instead of 200 milliseconds. Nothing else changed. But because partnerBankClient.sync() shares the same 20-thread pool as fraud scoring, notifications, and audit logging, those threads start piling up waiting on the slow partner call. Within minutes, all 20 threads are blocked. Fraud scoring stalls. Notifications stop going out. Audit logs fall behind. None of those systems are actually broken — they're just starved of threads by a completely unrelated integration.

This is the exact failure mode the Bulkhead Pattern exists to prevent.

Pattern: Bulkhead

The name comes from shipbuilding. A ship’s hull is divided into watertight compartments — bulkheads. If one compartment floods, the water stays there. It doesn’t sink the whole ship.

Applied to software, the idea is the same: give each integration or consumer its own isolated pool of resources — threads, connections, memory — so that if one is overwhelmed or slow, it can’t starve the others. No integration should be able to consume more than its fair share of a resource that others also depend on.

This is different from a Circuit Breaker, and the two are often confused. A Circuit Breaker decides when to stop calling a failing dependency. A Bulkhead decides how much of a shared resource that dependency is allowed to use in the first place — even while it’s still being called. In practice, you usually want both.

Applying Bulkhead at the Application Layer

The simplest fix to the code above is to stop sharing one executor across unrelated integrations. Give each one its own bounded pool:

@Configuration
public class IntegrationExecutorConfig {

    @Bean("fraudExecutor")
    public Executor fraudExecutor() {
        return Executors.newFixedThreadPool(5);
    }

    @Bean("notificationExecutor")
    public Executor notificationExecutor() {
        return Executors.newFixedThreadPool(5);
    }

    @Bean("partnerBankExecutor")
    public Executor partnerBankExecutor() {
        return Executors.newFixedThreadPool(3); // small on purpose — this one is slow and unreliable
    }
}

@Service
public class IntegrationDispatcher {

    @Async("fraudExecutor")
    public void scoreForFraud(TransactionEvent event) {
        fraudService.score(event);
    }

    @Async("notificationExecutor")
    public void sendNotification(TransactionEvent event) {
        notificationService.notify(event);
    }

    @Async("partnerBankExecutor")
    public void syncWithPartner(TransactionEvent event) {
        partnerBankClient.sync(event);
    }
}

Now, when the partner bank’s API slows down, only its own 3 threads back up. Fraud scoring and notifications keep running normally on their own pools — completely unaffected.

Resilience4j makes this even more explicit with a dedicated Bulkhead construct, which caps concurrent calls without you having to manage raw thread pools by hand:

@Bulkhead(name = "partnerBank", type = Bulkhead.Type.THREADPOOL)
public CompletableFuture<Void> syncWithPartner(TransactionEvent event) {
    return CompletableFuture.runAsync(() -> partnerBankClient.sync(event));
}

resilience4j.thread-pool-bulkhead:
  instances:
    partnerBank:
      max-thread-pool-size: 3
      core-thread-pool-size: 2
      queue-capacity: 10

If the queue fills up because the partner API is too slow, new calls fail fast instead of piling up indefinitely — the isolation holds even under sustained pressure.

Applying Bulkhead in Flink

The same problem shows up one layer down, inside stream processing itself — and this is where it gets interesting for CDC-style pipelines with multiple downstream consumers.

By default, Flink packs multiple operators from the same pipeline into the same task slot, sharing CPU and memory between them. That’s efficient — until one operator is heavier than the rest. A join against a partner-bank lookup table, for instance, can consume most of a shared slot’s resources, slowing down a completely unrelated map operator sitting in the same slot.

Flink’s version of Bulkhead is the slot sharing group. You explicitly assign heavy or unreliable operators to their own group, so they get isolated resources instead of competing with everything else:

DataStream<TransactionEvent> events = env
    .addSource(new FlinkKafkaConsumer<>("transactions", schema, props))
    .name("kafka-source");

DataStream<ScoredEvent> scored = events
    .map(new FraudScoringFunction())
    .name("fraud-scoring")
    .slotSharingGroup("fraud-group");

DataStream<TransactionEvent> partnerSynced = events
    .map(new PartnerBankSyncFunction())
    .name("partner-bank-sync")
    .slotSharingGroup("partner-bank-group"); // isolated — this one is slow

DataStream<TransactionEvent> audited = events
    .map(new AuditWriteFunction())
    .name("audit-write")
    .slotSharingGroup("audit-group");

env.setParallelism(4);

// give the slow, unreliable partner-bank operator fewer, dedicated resources
partnerSynced.getTransformation().setParallelism(2);

With this setup, if partner-bank-sync starts backpressuring — because the partner API can't keep up — that backpressure stays contained inside partner-bank-group. It doesn't propagate upstream into the fraud-scoring or audit-write operators, because they're running in physically separate task slots with their own resource budgets.

Without slot sharing groups, Flink’s default behavior would let all three operators share the same slots, and a slow partner-bank call would create backpressure that ripples backward through the whole job graph — eventually slowing down the Kafka source itself, and with it, every downstream consumer.

Why This Matters for Performance, Not Just Reliability

Bulkhead isolation isn’t only about surviving failure. Isolated resource pools also make performance predictable, which matters just as much in high-throughput integration platforms.

Bounded, isolated queues. Each integration gets a resource ceiling. You can reason about worst-case latency per integration instead of worst-case latency for the entire platform.

No cascading backpressure. In Flink specifically, a single slow join or slow sink no longer throttles the entire pipeline’s throughput — only the operators sharing its slot group feel the slowdown.

Capacity planning per integration. Because each pool is sized independently, you can give the fast, cheap operations (notifications, audit writes) small pools and the heavy, unreliable ones (partner APIs, fraud scoring against external services) their own larger — or deliberately smaller and stricter — allocation, based on their actual behavior rather than a single platform-wide guess.

Fail fast instead of fail slow. A bounded queue with a fast rejection is almost always better for the rest of the platform than an unbounded queue that lets one bad dependency quietly consume everything.

The Pattern, Stated Plainly

When multiple independent integrations share one pool of resources — threads, connections, task slots — a slowdown in any single one of them can silently degrade all the others. Give each integration its own bounded, isolated resource pool, sized for its own behavior, so that failure or slowness stays contained where it started.

That’s Bulkhead in one sentence, and it applies at every layer — application thread pools, Resilience4j configuration, and Flink slot sharing groups alike. The sign you need it is almost always the same: one dependency gets slow, and systems that have nothing to do with it start failing too. That’s not a coincidence. That’s a missing compartment wall.

Java #ApacheFlink #SpringBoot #SoftwareArchitecture #DesignPatterns #DataEngineering