How To Handle a Large Number of Uninterruptible Processes and Zombie Processes in a Linux System

High CPU usage waiting for I/O (referred to as iowait) is also one of the most common server performance issues. Today, we’ll look at a case of multi-process I/O and analyze this situation.

Process states

When iowait increases, processes are likely to remain in an uninterruptible state for an extended period due to the lack of hardware response. In the output of ps or top commands, you can observe that these processes are in the 'D' state, which stands for Uninterruptible Sleep. Speaking of process states, do you remember the different states a process can be in? Let's review them.

top and ps are the most commonly used tools for viewing process states, so let's start with the output from top. Below is an example of the output from the top command, where the 'S' column (Status column) indicates the state of the processes. In this example, you can see states like R, D, Z, S, and I. What do these states mean?

$ top
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
28961 root      20   0   43816   3148   4040 R   3.2  0.0   0:00.01 top
  620 root      20   0   37280  33676    908 D   0.3  0.4   0:00.01 app
    1 root      20   0  160072   9416   6752 S   0.0  0.1   0:37.64 systemd
 1896 root      20   0       0      0      0 Z   0.0  0.0   0:00.00 devapp
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.10 kthreadd
    4 root       0 -20       0      0      0 I   0.0  0.0   0:00.00 kworker/0:0H
    6 root       0 -20       0      0      0 I   0.0  0.0   0:00.00 mm_percpu_wq
    7 root      20   0       0      0      0 S   0.0  0.0   0:06.37 ksoftirqd/0

Let’s look at each state one by one:

• R (Running or Runnable): This means the process is in the CPU’s ready queue, either running or waiting to run.
• D (Disk Sleep): This indicates the process is in an uninterruptible sleep state, usually because it’s interacting with hardware. This interaction process cannot be interrupted by other processes or interrupts.
• Z (Zombie): If you have played the game “Plants vs. Zombies,” you should know what this means. It represents a zombie process, which means the process has actually ended, but its parent process has not yet reclaimed its resources (such as process descriptors, PID, etc.).
• S (Interruptible Sleep): This means the process is in a state of interruptible sleep, waiting for some event, and has been suspended by the system. When the event the process is waiting for occurs, it will wake up and enter the R state.
• I (Idle): This stands for idle and is used for kernel threads in uninterruptible sleep. As mentioned before, processes interacting with hardware are marked with D, but some kernel threads may actually have no load, so Idle is used to distinguish this situation. Note that processes in the D state will increase the average load, but processes in the I state will not.

Of course, the example above does not include all possible process states. In addition to the five states mentioned, there are two more:

• T or t (Stopped or Traced): This indicates the process is in a stopped or traced state. If a process receives a SIGSTOP signal, it will respond by entering the stopped state. Sending a SIGCONT signal will resume the process. If the process is started directly in the terminal, you need to use the fg command to bring it back to the foreground. When debugging a process with a debugger (such as gdb), the process will enter the traced state when interrupted by a breakpoint, which is a special kind of stopped state. However, you can track and control the process’s execution with the debugger as needed.
• X (Dead): This stands for dead, meaning the process has terminated, so you won’t see it in the output of top or ps.

With this understanding, let’s return to today’s topic. First, consider the uninterruptible state. This state ensures the consistency of process data and hardware status. Normally, processes stay in this state for a short period. Therefore, short-term uninterruptible processes can usually be ignored.

However, if there is a system or hardware failure, processes may remain in an uninterruptible state for a long time, potentially leading to many uninterruptible processes in the system. At this point, you should check if there are I/O performance issues.

Next, let’s look at zombie processes, a common issue in multi-process applications. Normally, when a process creates child processes, it should use system calls like wait() or waitpid() to wait for the child processes to end and reclaim their resources. Child processes will send a SIGCHLD signal to their parent process when they terminate, so the parent can also register a handler for the SIGCHLD signal to reclaim resources asynchronously.

If the parent process doesn’t do this, or if the child process executes too quickly and exits before the parent can handle its state, the child process becomes a zombie. In other words, the parent should be responsible for the child, ensuring proper management. If the parent fails to do this, “problematic” child processes can appear.

Usually, zombie processes don’t last long and will disappear after the parent reclaims their resources. If the parent process exits, the init process will reclaim and terminate the zombies.

If the parent process does not handle the termination of its children and remains running, the child processes will stay in the zombie state indefinitely. A large number of zombie processes can exhaust PID numbers, preventing the creation of new processes, so this situation must be avoided.

Case analysis

In the following section, I will use a multi-process application case to help you analyze issues related to a large number of uninterruptible (D state) and zombie (Z state) processes. This application is developed in C, and due to the complexity of its compilation and execution steps, I have packaged it into a Docker image for easy running.

First, I execute the following command to run the example application:

$ docker run --privileged --name=app -itd casestudy/app:iowait

Then, enter the ps command to verify that the example application has started correctly. If everything is fine, you should see the following output:

$ ps aux | grep /app
root      4009  0.0  0.0   4376  1008 pts/0    Ss+  05:51   0:00 /app
root      4287  0.6  0.4  37280 33660 pts/0    D+   05:54   0:00 /app
root      4288  0.6  0.4  37280 33668 pts/0    D+   05:54   0:00 /app

From this view, we can see that multiple app processes have been started, and their statuses are Ss+ and D+. Here, S indicates that the process is in an interruptible sleep state, and D indicates that it is in an uninterruptible sleep state, which we learned earlier. Don’t worry too much about the meanings of the s and + suffixes; you can check man ps for details. For now, remember that s indicates that the process is a session leader, and + indicates that it is part of the foreground process group.

Here, we have two new concepts: process groups and sessions. These concepts help manage groups of related processes, and their meanings are quite straightforward.

• Process Group: A collection of related processes where each child process is a member of the same group as its parent.
• Session: A collection of one or more process groups that share the same controlling terminal.

For example, when you log into a server via SSH, you open a controlling terminal (TTY), which corresponds to a session. The commands you run in the terminal and their child processes form process groups. Commands running in the foreground belong to the foreground process group, while those running in the background belong to the background process group.

With these concepts in mind, let’s use top to check the system's resource usage:

# Press the number 1 to switch to the usage of all CPUs, observe for a while, then press Ctrl+C to exit.
$ top
top - 05:56:23 up 17 days, 16:45,  2 users,  load average: 2.00, 1.68, 1.39
Tasks: 247 total,   1 running,  79 sleeping,   0 stopped, 115 zombie
%Cpu0  :  0.0 us,  0.7 sy,  0.0 ni, 38.9 id, 60.5 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu1  :  0.0 us,  0.7 sy,  0.0 ni,  4.7 id, 94.6 wa,  0.0 hi,  0.0 si,  0.0 st
...

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 4340 root      20   0   44676   4048   3432 R   0.3  0.0   0:00.05 top
 4345 root      20   0   37280  33624    860 D   0.3  0.0   0:00.01 app
 4344 root      20   0   37280  33624    860 D   0.3  0.4   0:00.01 app
    1 root      20   0  160072   9416   6752 S   0.0  0.1   0:38.59 systemd
...

Press 1 to switch to the CPU usage statistics for all CPUs. Observe the output for a while and then press Ctrl+C to exit.

Can you identify any issues from this? I have noticed four suspicious aspects.

Firstly, looking at the first line of average load (Load Average), the averages over the past 1 minute, 5 minutes, and 15 minutes are decreasing, indicating that the average load is increasing; with the 1-minute average load already reaching the number of CPUs in the system, it suggests that there is likely a performance bottleneck.

Next, on the second line under Tasks, there is 1 running process, but there are many zombie processes that keep increasing, indicating that some child processes are not being cleaned up upon termination.

Then, examining the CPU usage, both user CPU and system CPU are not high, but the iowait is 60.5% and 94.6% for the two CPUs respectively, which seems abnormal.

Finally, looking at each process, the highest CPU usage process is only at 0.3%, which doesn’t seem high; however, there are two processes in the D state, which might be waiting for I/O, but we cannot determine if they are causing the high iowait just from this information.

Summarizing these four issues, we can draw two clear conclusions:

1. The iowait is too high, causing the system’s average load to increase and even reach the number of system CPUs.
2. The number of zombie processes is constantly increasing, indicating that some programs are not properly cleaning up resources for child processes.

iowait analysis

I believe that when the term “increased iowait” is mentioned, your first instinct would be to check the system’s I/O status. This is also my usual approach. So, what tools can be used to check the system’s I/O status?

Here, I recommend the dstat command. Its advantage is that it can simultaneously display the usage of both CPU and I/O resources, making it convenient for comparative analysis.

Let’s run the dstat command in the terminal and observe the CPU and I/O usage:

# Output 10 sets of data at 1-second intervals.
$ dstat 1 10
You did not select any stats, using -cdngy by default.
--total-cpu-usage-- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai stl| read  writ| recv  send|  in   out | int   csw
  0   0  96   4   0|1219k  408k|   0     0 |   0     0 |  42   885
  0   0   2  98   0|  34M    0 | 198B  790B|   0     0 |  42   138
  0   0   0 100   0|  34M    0 |  66B  342B|   0     0 |  42   135
  0   0  84  16   0|5633k    0 |  66B  342B|   0     0 |  52   177
  0   3  39  58   0|  22M    0 |  66B  342B|   0     0 |  43   144
  0   0   0 100   0|  34M    0 | 200B  450B|   0     0 |  46   147
  0   0   2  98   0|  34M    0 |  66B  342B|   0     0 |  45   134
  0   0   0 100   0|  34M    0 |  66B  342B|   0     0 |  39   131
  0   0  83  17   0|5633k    0 |  66B  342B|   0     0 |  46   168
  0   3  39  59   0|  22M    0 |  66B  342B|   0     0 |  37   134

From the dstat output, we can see that whenever the iowait (wai) increases, the disk read requests (read) are significant. This indicates that the increase in iowait is related to disk read requests, and it’s likely caused by disk reads.

So, which process is reading the disk? If you remember from the previous section, we saw processes in an uninterruptible state in the top output. I find them very suspicious, so let’s try to analyze them.

Let’s continue in the terminal and run the top command to observe the processes in D state.

# Observe for a while and press Ctrl+C to stop.
$ top
...
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 4340 root      20   0   44676   4048   3432 R   0.3  0.0   0:00.05 top
 4345 root      20   0   37280  33624    860 D   0.3  0.0   0:00.01 app
 4344 root      20   0   37280  33624    860 D   0.3  0.4   0:00.01 app
...

From the output of top, find the PID of the processes in the D state. You can see that there are two processes in the D state with PIDs 4344 and 4345.

Next, let’s check the disk read and write activity of these processes. Don’t forget the tool to use. To examine the resource usage of a specific process, we can use our old friend pidstat, but this time, remember to add the -d parameter to output I/O usage.

For example, for process 4344, run the following pidstat command in the terminal, specifying the process ID with the -p 4344 parameter:

# -d displays I/O statistics, -p specifies the process ID, and it outputs 3 sets of data at 1-second intervals.
$ pidstat -d -p 4344 1 3
06:38:50      UID       PID   kB_rd/s   kB_wr/s kB_ccwr/s iodelay  Command
06:38:51        0      4344      0.00      0.00      0.00       0  app
06:38:52        0      4344      0.00      0.00      0.00       0  app
06:38:53        0      4344      0.00      0.00      0.00       0  app

In this output, kB_rd represents the number of KBs read per second, kB_wr represents the number of KBs written per second, and iodelay represents the I/O delay (in clock ticks). If they are all 0, it means there is no read or write activity at that moment, indicating that process 4344 is not causing the issue.

However, using the same method to analyze process 4345, you will find it also has no disk read or write activity.

So how do you determine which process is performing the disk read operations? We continue using pidstat, but this time without specifying a process ID to observe the I/O usage of all processes.

Run the following pidstat command in the terminal:

# Output data in intervals of 1 second, with 20 sets of data.
$ pidstat -d 1 20
...
06:48:46      UID       PID   kB_rd/s   kB_wr/s kB_ccwr/s iodelay  Command
06:48:47        0      4615      0.00      0.00      0.00       1  kworker/u4:1
06:48:47        0      6080  32768.00      0.00      0.00     170  app
06:48:47        0      6081  32768.00      0.00      0.00     184  app

06:48:47      UID       PID   kB_rd/s   kB_wr/s kB_ccwr/s iodelay  Command
06:48:48        0      6080      0.00      0.00      0.00     110  app

06:48:48      UID       PID   kB_rd/s   kB_wr/s kB_ccwr/s iodelay  Command
06:48:49        0      6081      0.00      0.00      0.00     191  app

06:48:49      UID       PID   kB_rd/s   kB_wr/s kB_ccwr/s iodelay  Command

06:48:50      UID       PID   kB_rd/s   kB_wr/s kB_ccwr/s iodelay  Command
06:48:51        0      6082  32768.00      0.00      0.00       0  app
06:48:51        0      6083  32768.00      0.00      0.00       0  app

06:48:51      UID       PID   kB_rd/s   kB_wr/s kB_ccwr/s iodelay  Command
06:48:52        0      6082  32768.00      0.00      0.00     184  app
06:48:52        0      6083  32768.00      0.00      0.00     175  app

06:48:52      UID       PID   kB_rd/s   kB_wr/s kB_ccwr/s iodelay  Command
06:48:53        0      6083      0.00      0.00      0.00     105  app
...

After observing for a while, you can see that it is indeed the app process that is performing disk reads, with 32 MB of data read per second. It looks like the issue is with the app process. But what I/O operations is the app process performing?

Here, we need to review the difference between user space and kernel space. For a process to access the disk, it must use system calls, so the next step is to find out what system calls the app process is making.

strace is a commonly used tool for tracing system calls made by a process. Therefore, we take the PID of the process from the pidstat output, for example, 6082, and then run the strace command in the terminal with the -p option to specify the PID:

$ strace -p 6082
strace: attach: ptrace(PTRACE_SEIZE, 6082): Operation not permitted

A strange error occurred here: the strace command failed, and the error reported is a lack of permissions. We should have been running all operations as the root user, so why is there a permissions issue? You might also think about how you would handle this situation.

When encountering such a problem, I usually start by checking the status of the process to see if it is normal. For example, continue in the terminal by running the ps command and use grep to find the 6082 process:

$ ps aux | grep 6082
root      6082  0.0  0.0      0     0 pts/0    Z+   13:43   0:00 [app] <defunct>

Sure enough, process 6082 has become a Z state, which means it is a zombie process. Zombie processes are those that have already exited, so we cannot continue analyzing their system calls. We will discuss how to handle zombie processes later; for now, let’s continue analyzing the iowait issue.

At this point, you should have noticed that the iowait issue persists, but tools like top and pidstat are no longer providing more information. This is when you should turn to dynamic tracing tools that are based on event records.

You can use perf top to see if there are any new discoveries. Alternatively, you can follow my approach and run perf record for a while (for example, 15 seconds) in the terminal, then press Ctrl+C to exit, and run perf report to view the report:

$ perf record -g
$ perf report

Next, find the app process we are interested in and press Enter to expand the call stack. You will get the following call graph:

In this diagram, the swapper is the scheduling process in the kernel, which you can ignore for now.

Looking at other information, you can see that the app is indeed using the system call sys_read() to read data. Furthermore, from new_sync_read and blkdev_direct_IO, we can see that the process is performing direct I/O operations on the disk, which means it bypasses the system cache and each read request directly accesses the disk. This explains the observed increase in iowait.

It turns out that the root cause is the app’s internal direct I/O operations!

The next step is to analyze the code to find where the direct read requests are coming from. In the source file app.c, you will see that it uses the O_DIRECT option to open the disk, thus bypassing the system cache and performing direct read and write operations on the disk.

open(disk, O_RDONLY|O_DIRECT|O_LARGEFILE, 0755)

Direct disk I/O is very friendly for I/O-sensitive applications (like database systems) because it allows you to control disk read and write operations directly within the application. However, in most cases, it is better to optimize disk I/O through the system cache. In other words, removing the O_DIRECT option is usually the way to go.

app-fix1.c is the modified file, and I have also packaged it into an image file. You can run the following command to start it:

$ docker run --privileged --name=app -itd casestudy/app:iowait-fix1

Finally, check using top:

$ top
top - 14:59:32 up 19 min,  1 user,  load average: 0.15, 0.07, 0.05
Tasks: 137 total,   1 running,  72 sleeping,   0 stopped,  12 zombie
%Cpu0  :  0.0 us,  1.7 sy,  0.0 ni, 98.0 id,  0.3 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu1  :  0.0 us,  1.3 sy,  0.0 ni, 98.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
...

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 3084 root      20   0       0      0      0 Z   1.3  0.0   0:00.04 app
 3085 root      20   0       0      0      0 Z   1.3  0.0   0:00.04 app
    1 root      20   0  159848   9120   6724 S   0.0  0.1   0:09.03 systemd
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kthreadd
    3 root      20   0       0      0      0 I   0.0  0.0   0:00.40 kworker/0:0
...

You will find that iowait is now very low, at just 0.3%, indicating that the recent changes have successfully fixed the high iowait issue. Mission accomplished!

However, don’t forget that zombie processes are still waiting for you. If you carefully observe the number of zombie processes, you will see that they are still continuously growing.

Zombie processes

Next, let’s address the issue of zombie processes. Since zombie processes appear because the parent process has not cleaned up the resources of the child processes, to resolve them, we need to find the parent process and address the issue there.

We’ve discussed how to find the parent process before. The simplest way to do this is to run the pstree command:

# -a option displays command-line arguments
# -p stands for PID
# -s specifies the parent process
$ pstree -aps 3084
systemd,1
  └─dockerd,15006 -H fd://
      └─docker-containe,15024 --config /var/run/docker/containerd/containerd.toml
          └─docker-containe,3991 -namespace moby -workdir...
              └─app,4009
                  └─(app,3084)

After running the command, you will find that process 3084 has a parent process with PID 4009, which is the app application.

So, the next step is to review the app application’s code to see if it correctly handles child processes' termination, such as checking for calls to wait() or waitpid(), or whether it has registered a SIGCHLD signal handler.

Now, let’s examine the source code in the file app-fix1.c to find the sections where child processes are created and cleaned up:

int status = 0;
  for (;;) {
    for (int i = 0; i < 2; i++) {
      if(fork()== 0) {
        sub_process();
      }
    }
    sleep(5);
  }

  while(wait(&status)>0);

Loops are inherently prone to errors, and can you spot the issue here? Although this code appears to call the wait() function to handle child process termination, it incorrectly places the wait() function outside of the for loop. This means that the wait() function is never actually called. By moving it inside the for loop, we can fix the problem.

The modified file is saved as app-fix2.c, and I have also packaged it into a Docker image. You can start it with the following command:

$ docker run --privileged --name=app -itd casestudy/app:iowait-fix2

After starting, check again with top one last time:

$ top
top - 15:00:44 up 20 min,  1 user,  load average: 0.05, 0.05, 0.04
Tasks: 125 total,   1 running,  72 sleeping,   0 stopped,   0 zombie
%Cpu0  :  0.0 us,  1.7 sy,  0.0 ni, 98.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu1  :  0.0 us,  1.3 sy,  0.0 ni, 98.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
...

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 3198 root      20   0    4376    840    780 S   0.3  0.0   0:00.01 app
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kthreadd
    3 root      20   0       0      0      0 I   0.0  0.0   0:00.41 kworker/0:0
...

Great! The zombie processes (Z state) are gone, and iowait is at 0. The issues have finally been resolved.

Conclusion

Today, I used a multi-process case to analyze the situation where the CPU utilization for waiting on I/O, or **iowait%**, increases.

Although this case showed that disk I/O caused the increase in iowait, high iowait does not always indicate a performance bottleneck in I/O. When the system is running only I/O-bound processes, iowait can be high, but this doesn’t necessarily mean that the disk's read/write operations are hitting a performance bottleneck.

Therefore, when you encounter high iowait, you should first use tools like dstat or pidstat to confirm whether the issue is indeed related to disk I/O, and then identify which processes are causing the I/O.

Processes waiting for I/O are generally in an uninterruptible state, so processes found in the D state (uninterruptible sleep) using the ps command are often suspect. However, in this case, after the I/O operation, the processes became zombie processes, so we could not use strace to directly analyze the system calls of these processes.

In such situations, we used the perf tool to analyze the system’s CPU clock events and ultimately discovered that direct I/O was the issue. Once we identified this, checking the relevant parts of the source code became straightforward.

The issue with zombie processes is relatively easier to diagnose. By using pstree to find the parent process, you can then examine the parent process’s code to check for wait() / waitpid() calls or the registration of a SIGCHLD signal handler.