The Day top Led Me From a Full Swap to a Process That Had Been Running for a Year

A step-by-step debugging story: how a routine memory check on a LAMP server uncovered a misconfigured shadow PHP pool — and, buried underneath it, a runaway refund job that had been burning a full CPU core since 2024.

I logged into a production server expecting nothing more than a routine memory check.

Thirty minutes later, I had uncovered two completely unrelated production issues: a hidden PHP-FPM installation quietly consuming gigabytes of RAM, and a Laravel refund process that had been burning an entire CPU core since 2024.

Neither issue had triggered alarms. Neither had caused an outage. Both had been sitting there in plain sight.

It all started with top.

top - 17:43:49 up 693 days,  4:18,  1 user,  load average: 1.09, 1.22, 1.16
MiB Mem :   7936.5 total,    327.1 free,   5011.9 used,   2597.6 buff/cache
MiB Swap:    512.0 total,      0.0 free,    512.0 used.    909.5 avail Mem

At first glance, nothing looked alarming.

The load average was modest. CPU usage was mostly idle.

But two numbers immediately caught my attention:

• Only 327 MB free, out of nearly 8 GB of RAM
• Swap: 100% used, with 0 MB free

A completely full swap isn’t necessarily an emergency, but it does remove your safety net. The next memory spike has nowhere to go except the OOM killer.

Then there was another number:

693 days of uptime.

Almost two years without a reboot. Plenty of time for configuration drift, forgotten services, and long-lived processes to quietly accumulate.

Rather than restarting services and hoping for the best, I decided to find out exactly where the memory was going.

Step 1: Find the memory hogs

ps aux --sort=-%mem | head

USER      PID     %CPU %MEM    RSS    COMMAND
mysql     3116146  2.1 10.0  819564  /usr/libexec/mariadbd
apache    914173   0.0  1.8  149864  php-fpm: pool preprod
apache    914175   0.0  1.8  146576  php-fpm: pool preprod
apache   1920280   0.0  1.7  145704  php-fpm: pool preprod
...

MariaDB consuming around 800 MB wasn’t unusual for this server.

What stood out was the long list of php-fpm workers, each using roughly 140 MB of memory. Even more interesting was the pool name attached to several of them:

**preprod**

Seeing a staging pool repeatedly appear among the biggest memory consumers on a production server felt worth investigating.

Step 2: Count the workers

ps aux | grep 'php-fpm: pool' | awk '{print $NF}' | sort | uniq -c

35 preprod
70 www

That meant:

• 70 www workers
• 35 preprod workers

A total of 105 PHP-FPM processes.

At roughly 140 MB each, that works out to around 15 GB of PHP worker memory on an 8 GB server.

Even allowing for shared memory pages, it was more than enough to explain why swap had been exhausted.

Step 3: Check the configured limits

grep -r "pm.max_children\|^pm " /etc/php-fpm.d/*.conf

pm = dynamic
pm.max_children = 50

Something immediately didn’t add up.

The production pool was capped at 50 workers, yet there were 70 running.

A simple reload brought the numbers back down.

systemctl reload php-fpm

The worker count dropped from 70 to 40.

One problem solved.

But another remained.

There was no preprod pool definition anywhere under /etc/php-fpm.d/, yet 35 preprod workers were still running.

Even stranger, I already knew this server had a supervisord-managed preprod process—but that service was only supposed to run two worker processes.

So where were these 35 PHP-FPM workers coming from?

Step 4: The pool that shouldn’t have existed

A process title like:

php-fpm: pool preprod

can only come from an actual PHP-FPM pool.

That meant another pool configuration existed somewhere.

grep -rl "\[preprod\]" /etc/ /opt/ 2>/dev/null

Eventually I found it.

/etc/opt/remi/php83/php-fpm.d/preprod.conf

The server wasn’t running one PHP installation.

It was running two.

PHP 8.3 had been installed separately through the Remi Software Collection, complete with its own PHP-FPM service, configuration directory, and pool definitions.

A quick check confirmed it.

systemctl list-units --type=service | grep -i php

php-fpm.service          active running
php83-php-fpm.service    active running

Two independent PHP-FPM services.

Two independent sets of pools.

Two separate configuration trees.

My earlier searches had completely missed the second installation because I had been looking in the obvious place.

Step 5: The misconfiguration

I opened the pool configuration.

grep -E "^pm|pm\.max_children" /etc/opt/remi/php83/php-fpm.d/preprod.conf

pm = dynamic
pm.max_children = 50
pm.start_servers = 5
pm.min_spare_servers = 5
pm.max_spare_servers = 35

There it was.

The staging pool had been configured almost identically to production.

The biggest culprit was this line:

pm.max_spare_servers = 35

PHP-FPM had effectively been instructed to keep 35 idle workers alive, whether traffic existed or not.

It looked very much like a production configuration that had been copied into staging and never right-sized.

I reduced it to something much more reasonable.

pm = dynamic
pm.max_children = 5
pm.start_servers = 2
pm.min_spare_servers = 1
pm.max_spare_servers = 2

Then reloaded the service.

systemctl reload php83-php-fpm

Worker counts immediately became:

2 preprod
11 www

Memory usage followed.

free -h

total   used    free    available
Mem:          7.8Gi   2.1Gi   3.3Gi   4.3Gi
Swap:         511Mi   487Mi   24Mi

Available memory jumped from only a few hundred megabytes to more than 4 GB.

Linux doesn’t automatically reclaim swapped pages just because RAM becomes available, so I reset swap manually.

swapoff -a && swapon -a

Swap: 511Mi   0B   511Mi

Clean.

At that point, I assumed the investigation was over.

It wasn’t.

Step 6: The part I wasn’t expecting

Out of habit, I ran one final sanity check.

ps aux --sort=-%mem | head

A few rows down sat this process.

root 2695670 99.9 5.2 424300 START:2024 TIME:920514:03 /usr/bin/php artisan initiate:refund

This wasn’t just another PHP process.

It was part of our refund pipeline.

And according to ps, it had been running continuously since 2024, consuming 99.9% CPU, with more than 920,000 CPU-minutes accumulated.

That’s roughly a year and a half of one CPU core.

The obvious question became:

Was it genuinely processing refunds?

Or was it simply stuck?

I checked its open file descriptors.

ls -l /proc/2695670/fd

One descriptor pointed to a Laravel log file that had already been deleted by log rotation months earlier.

Because the process still held the file open, the contents were still readable.

tail -n 200 /proc/2695670/fd/7

Next, I checked its network connections.

ss -tnp | grep 2695670

CLOSE-WAIT   127.0.0.1:xxxxx   127.0.0.1:6379
CLOSE-WAIT   127.0.0.1:xxxxx   127.0.0.1:6379
CLOSE-WAIT   127.0.0.1:xxxxx   127.0.0.1:6379

Three Redis connections.

All in CLOSE-WAIT.

Redis had closed its side of the connections long ago.

The PHP process never had.

Combined with the sustained 99.9% CPU usage, the evidence strongly suggested a queue worker that had lost its Redis connection and entered a tight retry loop instead of exiting or reconnecting cleanly.

After confirming there was no useful work taking place — no payment API traffic, no productive I/O, just a spinning core — I terminated it.

kill -TERM 2695670

It exited cleanly.

Load average dropped almost immediately.

One entire CPU core was finally available again.

What actually went wrong

Two completely unrelated issues had quietly coexisted on the same server.

First, a staging PHP-FPM pool from a separate PHP installation had been configured with production-level worker limits. Because it lived under /etc/opt/remi/php83/ and was managed by a separate systemd service, it remained invisible to my initial configuration checks while quietly reserving gigabytes of RAM.

Second, a refund-processing command had apparently lost its Redis connection sometime in 2024 and never recovered. Instead of failing or reconnecting, it appears to have entered a tight retry loop that consumed an entire CPU core for well over a year.

Neither issue would have been obvious from a single metric.

The memory investigation uncovered the hidden PHP installation.

A routine follow-up check exposed the CPU-bound zombie process.

Takeaways

• Run both ps aux --sort=-%mem and ps aux --sort=-%cpu. A process can be perfectly ordinary on one axis and a serious problem on the other.
• Always check for shadow installations. Multiple runtime versions often mean multiple services and multiple configuration trees.
• Don’t ignore START and cumulative TIME in ps. A process that's been running since last year deserves attention.
• **/proc/<pid>/fd is an underrated diagnostic tool.** Open file descriptors reveal what a process is actually doing—even when the original files have been deleted.
• A persistent CLOSE-WAIT socket isn't active work. It's often evidence that a process failed to clean up after the remote end disconnected.
• Long uptime is a smell, not a badge of honour. The longer a server runs, the more opportunities there are for configuration drift, forgotten services, and runaway processes to quietly accumulate.

What started as a simple question — “Why is swap full?” — ended with two production issues that had survived for months because nobody had been looking in the right places.

Monitoring tells you that something is wrong.

Investigation tells you why.

And in my experience, the best debugging sessions aren’t about knowing every Linux command. They’re about following the evidence until the system finally tells you its story.