NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver

Fixing nvidia-smi Failure on AWS EC2 (Amazon Linux 2023) with Blackwell GPUs

When launching GPU instances on AWS, one common issue is running nvidia-smi and seeing:

nvidia-smi: NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver.
Make sure that the latest NVIDIA driver is installed and running.

This happened while setting up a g7e.2xlarge EC2 instance running Amazon Linux 2023 with a Blackwell GPU. The system detected the GPU, but the driver failed to initialize.

Here’s what happened and how it was fixed.

The Problem

Running nvidia-smi produced either:

• a driver communication failure, or
• an empty GPU table even though /dev/nvidia0 existed.

Initial diagnostics showed the GPU was visible to the system:

lspci | grep -i nvidia

However, the NVIDIA kernel module was not properly loaded:

lsmod | grep nvidia

Checking kernel logs revealed the real issue:

NVRM: GPU 0000:XX:XX.X is not supported by the NVIDIA X.Y driver

The installed driver 560.35.03 did not support the Blackwell GPU (PCI ID 10de:2bb5).

Root Cause

The driver had been installed using the .run installer and was too old to support the Blackwell architecture. Because the GPU was unsupported, the driver failed to probe the device and could not register interrupts, leading to the nvidia-smi failure.

The Fix

The solution was to remove the old driver and install a newer version from NVIDIA’s CUDA repository.

1. Remove the old driver

sudo dkms remove nvidia/560.35.03 --all
sudo nvidia-uninstall --silent

Remove any leftover packages:

sudo dnf remove nvidia-fabric-manager -y

2. Add the CUDA repository

sudo dnf config-manager --add-repo \
https://developer.download.nvidia.com/compute/cuda/repos/amzn2023/x86_64/cuda-amzn2023.repo

3. Install the latest driver

sudo dnf install -y \
kmod-nvidia-open-dkms \
nvidia-driver \
nvidia-driver-cuda \
nvidia-fabricmanager

This installs:

• DKMS kernel module (auto rebuilds on kernel updates)
• NVIDIA userspace drivers
• CUDA utilities including nvidia-smi
• Fabric Manager for multi-GPU setups

4. Load the modules

sudo modprobe nvidia
sudo modprobe nvidia-uvm
sudo modprobe nvidia-modeset

5. Verify the installation

nvidia-smi

Expected output:

Driver Version: 595.45.04
CUDA Version: 13.2
GPU 0: NVIDIA RTX PRO 6000 Blackwell (97GB)

Important Notes

1. nvidia-smi not found after install

The binary is installed at:

/usr/bin/nvidia-smi

If your shell cannot find it:

hash -r

2. DKMS handles kernel updates

Because kmod-nvidia-open-dkms is used, the driver module is automatically rebuilt after kernel updates.

3. PCI INT A: no GSI warning

This warning sometimes appears on AWS Nitro instances due to ACPI interrupt routing. It is harmless because the driver falls back to MSI/MSI-X interrupts.

Key Takeaway

When running modern GPUs (like Blackwell) on cloud instances, driver compatibility is critical. Even if the GPU appears in lspci, an outdated driver can prevent the device from initializing.

Always install NVIDIA drivers from the CUDA repository instead of using .run installers on production systems. It simplifies updates, ensures compatibility, and avoids DKMS conflicts.

If you run into GPU initialization issues on EC2, checking driver support for your GPU architecture should be one of the first troubleshooting steps.