Building a JVM Health Check Task for Production Systems

This article reflects on a major issue that occurred in 2026 year.

At that time, a thread freeze happened in production, causing the entire project to hang. I had to handle the issue alone. After investigation, the root cause turned out to be an OutOfMemoryError (OOM).

The production environment had no monitoring system, so troubleshooting had to rely entirely on heap dump files and standard logs.

In real production systems, detecting abnormal JVM states early — such as thread hangs, memory leaks, and deadlocks — is crucial for maintaining service stability. Many teams embed lightweight health-check tasks directly into business services to periodically collect JVM metrics and record them in logs. This helps identify potential failures before they escalate into major outages.

In this article, we analyze a HealthCheckTask used in a real production project, exploring its design ideas, potential issues, and optimization strategies. The goal is to build a safe and efficient JVM health monitoring component suitable for production deployment.

1. Typical Design of a Health Check Task

1.1 Core Functions

A typical JVM health-check component performs several key tasks:

• Periodically collect thread states such as RUNNABLE, BLOCKED, and WAITING
• Monitor heap memory usage, GC count, and GC execution time
• Detect deadlocks and thread-hang symptoms (e.g., too many BLOCKED threads)
• Dump partial thread stack traces when suspicious behavior is detected

These metrics help engineers quickly identify performance bottlenecks and system anomalies.

1.2 Code Overview

Below is a simplified implementation of a JVM health check task.

@Component
@Slf4j
@ConditionalOnProperty(name = "health.check.enabled", havingValue = "true", matchIfMissing = true)
public class HealthCheckTask {

// Threshold configuration
    private static final int WARN_BLOCKED_THREAD = 5;
    private static final int WARN_TOTAL_THREAD = 500;
    // JMX Beans
    private final ThreadMXBean threadMXBean = ManagementFactory.getThreadMXBean();
    private final MemoryMXBean memoryMXBean = ManagementFactory.getMemoryMXBean();
    @Scheduled(fixedDelayString = "${health.check.interval:300000}")
    public void healthCheck() {
        long start = System.currentTimeMillis();
        try {
            // 1. Collect metrics
            ThreadStats stats = collectThreadStats();
            MemoryUsage heap = memoryMXBean.getHeapMemoryUsage();
            // 2. Log metrics
            log.info("Health check metrics: threads={}", stats);
            // 3. Detect suspicious behavior
            if (isSuspectHang(stats)) {
                suspectCount++;
                if (suspectCount >= CONTINUOUS_SUSPECT_LIMIT && canDump()) {
                    dumpSuspectThreads();
                    suspectCount = 0;
                }
            } else {
                suspectCount = 0;
            }
        } catch (Throwable t) {
            log.error("Health check execution error", t);
        } finally {
            long cost = System.currentTimeMillis() - start;
            if (cost > 100)
                log.warn("Health check execution time={}ms", cost);
        }
    }
}

The task uses Spring’s @Scheduled annotation to run periodically, with a default interval of 5 minutes, configurable through application properties.

Key Design Highlights

1. Switchable and configurable

• Controlled via @ConditionalOnProperty
• Execution interval configurable via property placeholders

2. Tiered logging

• Normal metrics → INFO
• Suspicious signals → WARN
• Sampling triggered → ERROR

3. Self-protection mechanism

• Thread dump only after 3 consecutive suspicious detections
• 10-minute cooldown period after a dump

4. Lightweight monitoring

• Regular checks do not capture thread stacks
• Sampling limits stack depth (5) and thread count (5)

This ensures minimal performance impact on production workloads.

2. Potential Runtime Issues

Even with careful design, several issues may still arise in real production environments.

2.1 Rigid Threshold Values

The code hardcodes thresholds such as:

WARN_BLOCKED_THREAD = 5
WARN_TOTAL_THREAD = 500

These values were tailored for a specific project whose core module uses Netty communication.

However, different applications have vastly different threading models:

• A Netty server may easily have thousands of threads
• A threshold of 500 threads may trigger constant warnings
• Temporary BLOCKED threads may occur normally (e.g., waiting for database connections)

Consequences

• Frequent false alarms
• Real issues may be buried in excessive warnings
• Unnecessary thread sampling

2.2 Thread Counting Inaccuracy

In the collectThreadStats() method, the total thread count is taken from:

ThreadInfo[] infos = threadMXBean.getThreadInfo(ids, 0);

If a thread terminates during retrieval, the corresponding element in the array becomes null.

Although the iteration skips null entries, the array length still counts them, causing a slightly inflated total thread count.

2.3 Precision Issues in Hang Detection

The hang detection logic:

if (s.total > WARN_TOTAL_THREAD && s.runnable < s.total * 0.1)

Potential issues include:

• Floating-point comparison may cause precision edge cases
• The 10% runnable threshold may not apply to all systems

For example:

A large thread pool with many threads in TIMED_WAITING (idle state) may still represent a healthy system.

2.4 Limitations of Deadlock Detection

threadMXBean.findDeadlockedThreads()

This detects JVM-level deadlocks caused by synchronized blocks.

However, it cannot detect deadlocks involving ReentrantLock from the java.util.concurrent package in some JVM environments.

If the application uses many explicit locks, deadlocks may go unnoticed.

2.5 Excessively Short Check Interval

If an operations engineer mistakenly configures:

health.check.interval = 1 second

Then every second the system will:

• Traverse all thread states
• Collect JVM metrics

In systems with tens of thousands of threads, this may significantly increase CPU overhead and affect production traffic.

3. Optimization Strategies

To address these issues, the health-check component can be enhanced in several ways.

3.1 Configurable Thresholds

Replace hardcoded values with configuration parameters:

@Value("${health.warn.blocked-thread:5}")
private int warnBlockedThread;

@Value("${health.warn.total-thread:500}")
private int warnTotalThread;

@Value("${health.warn.runnable-ratio:0.1}")
private double warnRunnableRatio;

This allows different teams to tune thresholds according to their system behavior.

3.2 Accurate Thread Counting

Use a more reliable method:

int totalThreads = threadMXBean.getThreadCount();

Alternatively, count non-null entries manually.

3.3 Improved Detection Logic

Avoid floating-point comparison:

s.runnable * 10 < s.total

Additional improvements:

• Consider TIMED_WAITING threads
• Trigger warnings only when runnable threads drop below a threshold and waiting threads dominate

3.4 Enhanced Deadlock Detection

Combine both APIs:

long[] deadlocked = threadMXBean.findDeadlockedThreads();

if (deadlocked == null) {
    deadlocked = threadMXBean.findMonitorDeadlockedThreads();
}

Together they provide broader deadlock coverage.

3.5 Minimum Interval Protection

Add a minimum interval safeguard:

if (interval < 10000) {
    log.warn("Health check interval too short, resetting to 10000ms");
}

Alternatively, enforce a minimum interval (e.g., 30 seconds) in the configuration center.

3.6 Safe Thread Stack Sampling

Before sampling:

• Check JVM system load
• Skip sampling if load is too high

Also:

• Execute sampling in a separate thread pool
• Avoid blocking the scheduled task thread

3.7 Enhanced Memory and GC Monitoring

Additional monitoring improvements:

• Trigger alerts when Old Gen usage exceeds 80%
• Track GC frequency and GC duration
• Monitor non-heap memory (Metaspace) usage

This helps detect classloader leaks and memory growth issues early.

4. Production Deployment Recommendations

1. Disable by default

Set:

health.check.enabled=false

Enable only on instances that require monitoring.

2. Tune thresholds carefully

Use stress testing environments to observe normal JVM metrics and define reasonable thresholds.

3. Integrate with monitoring systems

Send health-check logs to monitoring platforms such as:

• Prometheus
• Grafana

This enables visual dashboards and automated alerts.

4. Conduct periodic reviews

Regularly review:

• False alarms
• Missed alerts

Continuously refine thresholds and detection logic.

5. Conclusion

An effective JVM health-check system must balance two key goals:

1. Detect anomalies early
2. Minimize impact on production workloads

The HealthCheckTask discussed in this article already provides a solid foundation. With improvements such as:

• Configurable thresholds
• Accurate thread statistics
• Enhanced deadlock detection
• Safer sampling strategies

it can become a robust and reliable production monitoring component.

In real-world systems, there is no perfect monitoring script — only continuously evolving guardian processes that adapt to the system’s behavior.

Hopefully, this article provides useful insights for designing and optimizing your own JVM health monitoring solution.

🔖 Thanks for reading.

• If you enjoyed this article, please consider giving it a clap.👏
• I would appreciate hearing your thoughts in the comments below! 💭
• Follow me for ongoing learning and connection!🔔