K3s Nodes Stuck in NotReady After Reboot? It’s Probably Clock Skew.

If your K3s nodes go NotReady for 10–15 minutes after every reboot with x509 certificate errors like this:

x509: certificate has expired or is not yet valid:
  current time 2026-04-01T09:21:14Z is before 2026-04-01T16:13:29Z

The problem isn’t expired certificates — it’s your system clock.

What’s Happening

On boot, K3s starts before NTP has corrected the system clock. If the hardware clock is off (common with VMs or when RTC stores local time instead of UTC), K3s generates certificates with a Not Before timestamp based on the wrong time.

For example, if the hardware clock is set to local time (UTC+8) but the OS interprets it as UTC, K3s might generate a certificate with Not Before: 16:13 UTC.

Minutes later, NTP corrects the system clock to the actual time: 09:21 UTC. Now the certificate's Not Before is 7 hours in the future. Any TLS handshake that checks the certificate validity — which is all of them — sees a cert that isn't valid yet and rejects it with Unauthorized.

K3s will eventually detect the bad certs, restart itself, and regenerate them with the correct time. But this self-healing cycle takes 10–15 minutes — during which the node is NotReady and spamming auth errors.

The Fix

Force K3s to wait for NTP sync before starting. On each node:

sudo mkdir -p /etc/systemd/system/k3s.service.d
sudo tee /etc/systemd/system/k3s.service.d/wait-for-ntp.conf <<'EOF'
[Unit]
After=time-sync.target
Wants=time-sync.target

[Service]
ExecStartPre=/bin/sh -c 'until timedatectl show -p NTPSynchronized --value | grep -q yes; do echo "Waiting for NTP sync..."; sleep 1; done'
EOF

sudo systemctl daemon-reload

This creates a systemd drop-in override that gets merged with the existing k3s.service unit. The After= and Wants= directives ensure K3s is ordered after the time sync target, and the ExecStartPre loop blocks startup until NTP has actually synchronized — not just started.

After a daemon-reload, confirm systemd picked it up:

sudo systemctl show k3s.service --property=DropInPaths

Next reboot, K3s will wait for a valid clock before generating certificates. Recovery time drops from ~15 minutes to under a minute or two.

Glossary

• VM (Virtual Machine) — A software-emulated computer running on top of a physical host. VMs often have less reliable hardware clocks compared to bare metal since the hypervisor virtualizes the clock.
• RTC (Real-Time Clock) — A battery-backed hardware clock that keeps time even when the machine is powered off. On boot, the OS reads the RTC to set the initial system time before NTP takes over.