The Pain of Heap Dump Analysis: Why Your Team Spends Days, Not Minutes, on OutOfMemoryError

One of the most common causes of Java application crashes is OutOfMemoryError. Sadly, in many organizations, this problem can take days to solve, while administrators experiment with tweaking the size of the Java heap space or even adding more physical memory. Meanwhile, developers work through screeds of code searching for memory leaks and wasted memory.

How do some teams manage to solve OutOfMemoryErrors in minutes? In this article, we’ll look at some of the reasons troubleshooters may take too long to solve errors, and how to speed up the debugging process.

We’ll find out how to quickly identify the right solution, which could be improving the code, changing the JVM configuration, or adding more physical memory.

Why Do Some Teams Spend Too Long Solving OutOfMemoryErrors?

Here are some of the issues that slow down this type of problem solving:

• We don’t know how to establish the type of OutOfMemoryError, and what it relates to;
• We don’t know how to take a heap dump;
• We don’t have a suitable tool for analyzing heap dumps.
• We don’t have a clear problem-solving strategy.

Establishing the Error Type

Many OutOfMemory errors are heap-related, and can be solved by analyzing a Java heap dump.

However, this is not always the case. The JVM memory has several different areas, each with its own purpose. This is illustrated in the diagram below.

Fig: JVM Memory Model

OutOfMemoryErrors may relate to non-heap spaces, such as the Metaspace or the Direct Buffer Space. When the JVM encounters an OutOfMemoryError, it prints a description of the type of error followed by the stack trace showing where the error occurred. Reading the error message carefully to identify the type of error is the first step in troubleshooting the problem. In fact, there are nine types of OutOfMemoryErrors. For a description of each, and hints on how to identify the cause, see Types of OutOfMemoryError.

Taking a Dump of the Java Heap Space

Java heap dumps are captured into a file, which will be in binary format. Many Java heap space analyzer tools require the file to have the extension of .hprof.

There are several different ways to take a heap dump. These are described here: Ways to Capture a Heap Dump.

One example is to use the *jmap* utility, which is supplied with the JDK and run from the command line. The command looks like this:

jmap -dump:format=b,file=<file path> <pid>

<file-path> is the path of the file that will contain the heap dump, and <pid> is the process ID of the application.

Heap dumps may be very large, since they contain the entire contents of the application’s heap space. If the application processes sensitive information such as credit card numbers, the heap dump is likely to contain this data. Heap dumps should always be treated as confidential and protected from hackers.

Tools For Analyzing the Heap Space

It would be impractical to analyze a heap dump manually, since it is a very large binary file.

Several excellent free tools are available that make heap dump analysis simple. The all-purpose troubleshooting and monitoring tool VisualVM includes basic heap dump analysis. More specialized tools include Eclipse MAT and HeapHero. In this article, we’ll include examples from HeapHero, which has several additional advantages, including machine learning algorithms that can often instantly diagnose the problem.

For more details, you may like to read this article: Choosing the Right Memory Dump Analyzer.

Solving OutOfMemoryErrors: A Strategy

Let’s look at how we can go about using a heap space analyzer to solve OutOfMemoryErrors in Java. We’ll use HeapHero in the examples, but the steps would be the same for other tools such as Eclipse MAT. We would first need to take a heap dump. For MAT, the file extension must be .hprof.

1. Look For Any Suggestions Made by the Analyzer

Heaphero provides a detailed report. If the ML algorithms detected any possible issues, it displays them at the front of the report. The suggestions may look like this:

Fig: HeapHero ML Suggestions

Eclipse MAT displays a Leak Suspect chart on the initial screen.

2. Establish The Largest Objects

In nearly all cases, the cause of the OutOfMemoryError will be related to one of the top two or three largest objects in memory. The HeapHero report contains a list of all objects in memory, sorted from largest to smallest. The list looks like this:

Fig: List Of Largest Objects Included in the HeapHero Report

At this level, the largest object is often a thread, and in the next step we’ll learn how to drill down from this to find the actual culprit.

Shallow heap is the amount of memory used by the object itself, whereas retained heap is the amount of memory occupied by the object and all its children.

3. Drill Down the Dominator Tree

The dominator tree is a theoretical way of looking recursively through objects to find out what object created them, and what objects they created.

Start with the largest object in memory, and follow the dominator tree to see which of its children are holding the most memory. In HeapHero, you can do this by clicking ‘More’, and choosing ‘Outgoing References’. You can also expand the arrow to the right of the object’s entry on the Largest Object report. Again, the child objects will be sorted from largest to smallest, and the culprit is likely to be near the top of the list.

We can expand each object further to see its current contents in memory.

We can now ask ourselves questions such as ‘Why is this object so large’, and ‘Does it need to be so big?’ We can start looking at the code to see whether there is a memory leak, or any other issues.

4. Explore Upwards Through the Dominator Tree

This exercise tells us what objects are holding references to the large objects, and preventing them from being garbage collected. We will often find that the problem is caused by references not being released when they should be. References are released either when the block of code that contains them completes, or when an object is explicitly set to null. If an object is defined in the wrong scope, it may stay in memory even when it’s no longer needed.

We can look at the code and make sure data is not held in memory for longer than it should be.

5. Check for Wasted Memory

If the previous steps haven’t uncovered a memory leak or other buggy code, we’d next look for wasted memory. Most applications waste an incredible amount of memory due to poor coding practices.

HeapHero includes a Wasted Memory Report, and it’s well worth looking at this and making improvements. The diagram below shows an overview of wasted memory within an application.

Fig: HeapHero’s Wasted Memory Report

This is followed by individual reports listing the objects in each category.

If none of these strategies have uncovered memory leaks or wasted memory, we may need to adjust the heap space size or add more physical memory.

To see all of this in action, you may like to watch this video: How to Analyze a Heap Dump.

Conclusion

Analyzing the Java heap space to solve OutOfMemoryErrors need not be time-consuming or difficult.

With tools such as HeapHero or Eclipse MAT, we can find the problem, work out the best solution and get the system back up and running with minimal delay.