How supplyAsync() without an Executor Froze Our Internal Common Pool

On a Thursday afternoon, our real-time notification engine (which delivers email notifications, push notifications, and SMS alerts to millions of active users) silently stopped working.

There were no crashes. No JVM OutOfMemory errors. No database outages.

But at 3:15 PM, our customer onboarding metrics dipped to zero. New users registering on our platform were not receiving their verification OTPs. Transaction confirmations were delayed by hours. The queue depth on our notification processor grew to 120,000 pending jobs.

Our APM logs showed:

[2026-07-03 15:15:02.991] WARN  [ForkJoinPool.commonPool-worker-3] c.c.n.s.EmailDeliveryService: Slow email SMTP dispatch took 30005ms
[2026-07-03 15:15:33.001] WARN  [ForkJoinPool.commonPool-worker-1] c.c.n.s.SmsDeliveryService: SMS gateway handshake delayed by 15000ms

A quick check revealed that the application was completely responsive to health checks, but any process requiring asynchronous background execution was indefinitely hanging.

The Day the Notification Engine Went Dead Silent

When our systems team looked at the threads, they noticed a strange pattern. The CPU utilization was sitting at 1% on the JVM, and yet all thread dumps pointed to the same bottleneck:

"ForkJoinPool.commonPool-worker-1" #42 daemon prio=5 os_prio=31 cpu=112.44ms elapsed=1820s tid=0x00007f87c98c9000 nid=0x6703 waiting on condition  [0x000070000a6c2000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x0000000715aa31b8> (a java.util.concurrent.ForkJoinPool)
    at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824)
    at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693)
    at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)

The entire ForkJoinPool.commonPool was completely saturated. Almost all worker threads were locked in a TIMED_WAITING state, holding up tasks across multiple services. Our SMS alerts, image generation pipelines, audit logs, and analytics tracking were all frozen. It was a classic cascade failure.

Digging Through the Thread Logs

When a service hangs, you usually check for network splits or database locks. We spent the first 20 minutes chasing down those leads.

1. The Network Packet Drop Hypothesis

Because the SMTP server and the SMS gateway are external, we initially suspected network latency or firewall packet drops. We verified outgoing connections using tcpdump and confirmed that the packets were routing successfully:

sudo tcpdump -i any 'port 25 or port 587'

The connections were slow, but they were not dropping. Network bandwidth and DNS resolution times were normal.

2. The Database Connection Shortage Hypothesis

We checked if the threads in the common pool were blocked waiting for database connections to log the notification outcomes:

SELECT pid, query, state, age(clock_timestamp(), query_start) 
FROM pg_stat_activity 
WHERE state != 'idle';

There were no long-running queries or locked tables. The threads were not blocked at the database level.

Under the Hood: The Shared Thread Pool Trap

To understand the cause of the freeze, we have to look under the hood of Java’s async concurrency utilities, specifically CompletableFuture.

When developers want to run a task asynchronously in Java, the most common pattern is:

CompletableFuture.supplyAsync(() -> {
    return fetchUserData(userId);
});

What most developers overlook is the default executor used by CompletableFuture when no executor is explicitly supplied:

public static <U> CompletableFuture<U> supplyAsync(Supplier<U> supplier) {
    return asyncSupplyStage(asyncPool, supplier);
}

Here, asyncPool points directly to the static JVM-wide **ForkJoinPool.commonPool()**.

The Vulnerability of a Shared Pool

The ForkJoinPool.commonPool() is designed for CPU-bound computation tasks using the Work-Stealing Algorithm. By default, Java sizes this pool to:

Number of target processors−1Number of target processors−1

On our production container instances (which have 4 vCPUs allocated), the common pool was initialized with exactly 3 worker threads.

The disaster unfolded when a junior developer added an asynchronous email sender:

public CompletableFuture<Void> sendVerificationEmail(User user) {
    return CompletableFuture.supplyAsync(() -> {
        // High latency I/O call: connects to external SMTP server
        emailClient.send(user.getEmail(), "Verify your email", TEMPLATE_ID);
        return null;
    });
}

Because emailClient.send() is a synchronous, blocking network call, when 3 registration requests came in concurrently, they consumed all 3 worker threads in the common pool.

At that point:

• The common pool was completely starved of threads.
• Any other part of the application that relied on CompletableFuture (without an explicit executor) or parallel streams (myList.parallelStream().map(...)) was queued indefinitely.
• Because the SMTP provider took up to 30 seconds to fail, the entire JVM was essentially blocked for all asynchronous activities.

The Fix: Setting Up Proper Segregation

To solve this, we implemented strict pool segregation. We configured dedicated executors for our asynchronous tasks, ensuring that blocking I/O calls are isolated from CPU-bound computation and other services.

1. Configure a Dedicated Executor for Notifications

We created a custom ThreadPoolTaskExecutor sized specifically for blocking I/O:

package com.company.config;

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor;

import java.util.concurrent.Executor;
import java.util.concurrent.ThreadPoolExecutor;

@Configuration
public class AsyncExecutorConfig {

    @Bean(name = "notificationExecutor")
    public Executor notificationExecutor() {
        ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();

        // Match core size to expected concurrent I/O connections
        executor.setCorePoolSize(50);

        // Allow dynamic scaling up to 150 threads under heavy loads
        executor.setMaxPoolSize(150);

        // Prevent out-of-memory errors by bounding the task queue
        executor.setQueueCapacity(1000);

        executor.setThreadNamePrefix("notification-io-");

        // Reject strategy: run the task on the caller thread if the queue is saturated
        executor.setRejectedExecutionHandler(new ThreadPoolExecutor.CallerRunsPolicy());

        executor.initialize();
        return executor;
    }
}

2. Supply the Executor Explicitly to CompletableFuture

We modified the service code to ensure the CompletableFuture tasks are routed to our dedicated executor instead of the JVM common pool:

package com.company.service;

import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.stereotype.Service;

import java.util.concurrent.CompletableFuture;
import java.util.concurrent.Executor;

@Service
public class EmailService {

    private final EmailClient emailClient;
    private final Executor notificationExecutor;

    public EmailService(EmailClient emailClient, @Qualifier("notificationExecutor") Executor notificationExecutor) {
        this.emailClient = emailClient;
        this.notificationExecutor = notificationExecutor;
    }

    public CompletableFuture<Void> sendVerificationEmail(User user) {
        // Explicitly supply the custom executor
        return CompletableFuture.supplyAsync(() -> {
            emailClient.send(user.getEmail(), "Verify your email", "TEMPLATE_123");
            return null;
        }, notificationExecutor); // Segregated executing thread
    }
}

With this change:

• The 50 to 150 threads in notification-io- handle all slow SMTP and SMS tasks.
• The ForkJoinPool.commonPool() is completely untouched and remains available for internal JVM optimizations, class loading, stream processing, and parallel operations.
• Saturated queues will gracefully degrade by running tasks on the caller thread (CallerRunsPolicy), preventing tasks from being lost silently.

Did It Actually Work? The Post-Mortem Benchmark

We simulated a complete SMTP outage (30-second handshake delay) and ran load tests using 500 concurrent registration requests:

Comparative Metrics

Metric                           Using Default ForkJoinPool.commonPool()    Using Segregated notificationExecutor
-------------------------------------------------------------------------------------------------------------------
Max Common Pool Threads          3 (on 4 vCPU system)                       3 (Unchanged, completely idle)
Active I/O Threads               3                                          150
OTP Verification Delivery Lag    >15 minutes (or lost)                      Sub-second (Queue buffered)
JVM Parallel Stream Throughput   Dropped to 0 (Blocked)                     Unaffected (Normal processing)
Application Crash / Deadlock     Silent freeze                              Healthy operation

By switching from the shared common pool to a dedicated thread pool for blocking calls, we preserved system integrity during downstream provider outages. Parallel processing and other system tasks continued without a millisecond of lag.

What I Tell My Tech Leads

The ForkJoinPool.commonPool() is a shared global resource. Any library or code block running in the same JVM instance shares these same worker threads. If you block a thread in this pool, you block it for the entire application.

As a general design rule, never perform blocking I/O (database, file system, HTTP network calls) inside the common pool. If a method requires asynchronous execution, ensure it accepts a custom Executor as a parameter. Allowing CompletableFuture to default to commonPool() is a ticking time bomb in microservice architectures.

The Retrospective Cheat Sheet

Category         Detail
------------------------------------------------------------------------------------------------------------------------------------
Symptom          Asynchronous email/SMS deliveries ceased completely; background jobs hung; OTP verification delayed by minutes.
Root Cause       Async tasks were scheduled via CompletableFuture.supplyAsync() without an executor, saturating the static ForkJoinPool.commonPool().
The Fix          Created a dedicated ThreadPoolTaskExecutor (notificationExecutor) and passed it explicitly to supplyAsync().
The Result       Notification I/O execution was isolated. Downstream latency had zero impact on CPU-bound processes or shared common pool operations.

Want to dive deeper into system design, database internals, and actual engineering war stories? Join 14+ senior engineers for weekly post-mortems and architectural deep dives by subscribing below.