Escaping the Infinite NFSv4 Grace Period

Every infrastructure engineer knows the feeling: the post-upgrade calm that is just a little too quiet.

It started as a standard morning maintenance window. I was upgrading our production Ceph cluster from 19.2.3 to 20.2.1. By all standard metrics, it was a textbook success. The dashboards were entirely green. The VMs backing their storage via RBD were perfectly reachable, and a quick fio benchmark showed performance was exactly where it needed to be.

I was just about to close the maintenance window when the first ticket came in: “The NFS shares are down.”

In our architecture, we use CephFS exposed to customers through NFS-Ganesha gateways. The symptom was bizarre. Clients could attempt to list directories, but the terminal would just hang indefinitely. No timeouts, no connection refused — just an endless freeze.

The Ceph orchestrator reported all NFS daemons were healthy. The network showed no dropped packets. The dashboard was lying. It was time to dig deeper.

Isolating the Fault Domain

Before falling down a rabbit hole, I needed to divide the architecture and isolate the fault domain. Was the underlying CephFS file system broken, or was the issue sitting strictly at the NFS gateway layer?

To test this, I bypassed the NFS gateways entirely. I took a test client and mounted the shares directly using the native CephFS kernel module.

The native mounts worked flawlessly. Files were accessible, read/write speeds were great, and there were no hangs. This confirmed the CephFS backend was perfectly healthy. The problem was isolated entirely to the NFS-Ganesha daemons.

The Smoking Gun: Packet Analysis

Now that I knew where to look, I needed to know what was failing. I fired up tcpdump between a hanging client and the Ceph NFS gateway and caught this exchange:

NFS request xid ... getattr fh 0,2/53 NFS reply xid ... reply ok 96 getattr ERROR: unk 10013

The client was asking for file attributes (getattr), and the server was immediately rejecting it with error code 10013.

In the NFSv4 protocol, 10013 translates to NFS4ERR_GRACE.

When an NFSv4 server reboots, it enters a “Recovery Grace Period.” During this window, it temporarily rejects new file access requests to give existing clients time to reclaim their previous locks. But a grace period is supposed to last 90 seconds. Ours was lasting forever.

The Root Cause: Distributed State

Because Ceph is highly available, NFS-Ganesha daemons coordinate their grace periods using a shared database stored inside a RADOS pool.

When the upgrade from 19.2.3 to 20.2.1 rolled through, the NFS daemons were restarted. However, a phantom record of the old daemons remained stuck in the RADOS database, flagged as needing a grace period (N).

The newly upgraded daemons booted up, checked the RADOS state, and essentially said: “We cannot end the grace period until those old nodes check in.” Because those old nodes no longer existed, the cluster was deadlocked in a permanent grace period.

The Surgical Fix

Fixing this required interacting directly with the RADOS grace database, but because Ceph containerizes its daemons with strict least-privilege keyrings, I couldn’t just run the command from the host.

Here is how we broke the deadlock:

1. Entering the Container Shell I used cephadm to drop into the specific, isolated container of the active NFS daemon.

2. Parsing the Credentials I extracted the specific RADOS pool (.nfs), namespace (cephnfs), and User ID from the daemon's ganesha.conf.

3. Auditing the Database Running the internal Ganesha tool, I queried the cluster state: ganesha-rados-grace -p .nfs -n cephnfs -u <UserId> dump

The output revealed the ghost nodes holding the N (Needs Grace) flag, forcing the active nodes into an E (Enforcing) state.

4. Lifting the Blockade I issued the command to forcibly clear the database: ganesha-rados-grace -p .nfs -n cephnfs -u <UserId> lift

I ran dump one last time. The flags were gone.

Instantly, across the environment, the hanging client terminals unfroze. The queued getattr requests succeeded, and data started flowing again.

The Takeaway

A green dashboard only tells you that the services are running; it doesn’t tell you what they are thinking.

Upgrading distributed systems often introduces edge cases where state outlives the daemons themselves. Knowing how to isolate a fault domain, drop down to the packet level, translate a protocol error (10013), and surgically clear a distributed lock is what separates following a runbook from true infrastructure engineering.

Has anyone else run into infinite grace periods or ghost states during a major storage upgrade? I’d love to hear your debugging war stories in the comments!