When you run nvidia-smi on a Linux server and receive the error NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver, it usually means that the NVIDIA user-space utility cannot communicate with the NVIDIA kernel driver. The GPU may be physically installed, but its driver module might not be loaded, might have failed to build after a Linux kernel update, or might be blocked by Secure Boot.
This issue can affect Linux GPU servers running AI workloads, machine learning applications, CUDA, GPU-accelerated computing and other workloads that depend on NVIDIA hardware.
In this guide, we explain how to diagnose and fix the NVIDIA-SMI communication error on Ubuntu, Debian, Rocky Linux and other compatible Linux distributions. You will learn how to check GPU detection, inspect NVIDIA kernel modules, repair DKMS build failures, resolve Secure Boot conflicts and verify that the GPU driver is working correctly.
1. What Does the NVIDIA-SMI Communication Error Mean?
nvidia-smi (NVIDIA System Management Interface) is a command-line utility used to monitor and manage NVIDIA GPUs on Linux systems. It reports information such as GPU utilisation, memory usage, temperature, power consumption, running processes and driver versions.
When the command runs successfully, it communicates with the NVIDIA Management Library (NVML), which interacts with the NVIDIA kernel driver to retrieve GPU information.
However, when the required kernel module is unavailable or cannot communicate with the GPU, the command may return this error:
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver.
Make sure that the latest NVIDIA driver is installed and running.
This does not automatically mean that the GPU hardware is faulty or that you need to install a newer driver.
Linux uses kernel modules to allow the operating system to communicate with hardware devices. The NVIDIA driver includes modules such as nvidia.ko, nvidia-uvm.ko and nvidia-modeset.ko. If the main NVIDIA module is missing, incompatible with the running kernel or prevented from loading, nvidia-smi cannot access the GPU correctly.
Common causes include:
Missing NVIDIA driver: The correct NVIDIA driver is not installed or the installation did not complete successfully.
Linux kernel update: A kernel upgrade has left the NVIDIA kernel module unavailable for the newly running kernel.
DKMS build failure: Dynamic Kernel Module Support (DKMS) could not compile the NVIDIA module for the current kernel.
Secure Boot restrictions: Secure Boot prevents an unsigned or untrusted NVIDIA kernel module from loading.
Driver and kernel version mismatch: The installed driver components do not match the module currently loaded into the kernel.
Nouveau driver conflict: The open-source Nouveau driver is interfering with the NVIDIA driver.
GPU detection or passthrough issues: The Linux operating system cannot access the physical GPU or a GPU assigned to a virtual machine.
Identifying the actual cause before reinstalling the driver helps avoid unnecessary changes to a working Linux server.
2. Check Whether Linux Detects the NVIDIA GPU
Before changing NVIDIA driver packages, confirm that the operating system can detect the GPU.
Open a terminal and run:
lspci | grep -i nvidia
This command lists PCI devices associated with NVIDIA hardware.
A detected GPU may return output similar to:
01:00.0 VGA compatible controller: NVIDIA Corporation ...
01:00.1 Audio device: NVIDIA Corporation ...
The exact device name and PCI address depend on the GPU model and server configuration.
What if the command returns no output?
If lspci does not list an NVIDIA device, the problem may be below the driver level. Reinstalling the NVIDIA driver will not fix hardware that the operating system cannot detect.
Check the following:
Confirm that the GPU is physically installed and properly connected.
Check PCIe power connections and the GPU's power supply.
Verify that the server BIOS or UEFI detects the GPU.
Check PCIe slot configuration and firmware settings.
If you are using a virtual machine, confirm that PCI passthrough or the required virtual GPU assignment is configured correctly.
You can also inspect PCI devices with:
sudo lspci -nn
Important: On a GPU virtual machine, the host and guest have different hardware visibility. A GPU being visible on the host does not guarantee that it has been correctly assigned to the guest operating system.
Once Linux detects the NVIDIA GPU, continue with the driver checks.
3. Check the NVIDIA Driver and Kernel Module Status
The next step is to determine whether the NVIDIA kernel module is installed and loaded.
Run:
lsmod | grep nvidia
The lsmod command displays kernel modules currently loaded into the Linux kernel.
A working NVIDIA installation may show entries similar to:
nvidia_uvm
nvidia_drm
nvidia_modeset
nvidia
The exact modules depend on the driver configuration and enabled features. If the command returns no output, the NVIDIA modules may not be loaded.
Check the installed NVIDIA driver
Run:
modinfo nvidia
This command displays information about the NVIDIA kernel module, including its filename, version and supported aliases.
If you receive:
modinfo: ERROR: Module nvidia not found.
The module may not be installed for the current kernel, or the module installation may have failed.
Next, check the running Linux kernel:
uname -r
Then inspect the kernel messages for NVIDIA-related errors:
sudo dmesg -T | grep -iE 'nvidia|nvrm|nouveau'
You can also use:
sudo journalctl -k -b | grep -iE 'nvidia|nvrm|nouveau'
These commands help identify issues such as failed module loading, unsupported devices, Secure Boot restrictions and driver version mismatches.
Why this matters: The NVIDIA kernel module must be compatible with the kernel currently running on the server. Finding the error in the kernel logs is often more useful than immediately reinstalling the entire driver.
4. Fix NVIDIA Driver Issues After a Linux Kernel Update
A Linux kernel update is one of the common reasons an NVIDIA driver suddenly stops working.
For example, an administrator may update Ubuntu or Rocky Linux and reboot the server. The system then starts using a newer kernel, but the NVIDIA kernel module has not been installed or rebuilt for that version. As a result, nvidia-smi can no longer communicate with the driver.
Step 1: Check the running kernel version
Run:
uname -r
Step 2: Check whether the NVIDIA module exists for that kernel
Run:
modinfo -F filename nvidia
If the module is missing, or its installation does not match the running kernel, check the relevant driver packages and DKMS status.
Step 3: Install the matching kernel headers
On Ubuntu and Debian systems:
sudo apt update
sudo apt install linux-headers-$(uname -r)
On Rocky Linux, AlmaLinux and other compatible RHEL-based distributions:
sudo dnf install kernel-devel-$(uname -r) kernel-headers-$(uname -r)
The installed development files must match the kernel being used. On distributions with separate kernel package flavours, ensure that the appropriate headers or development package is installed.
Step 4: Rebuild the NVIDIA kernel module
If the NVIDIA driver is registered with DKMS, run:
sudo dkms status
If an NVIDIA module is registered but has not been built for the current kernel, rebuild it:
sudo dkms autoinstall
After the build completes, try loading the driver:
sudo modprobe nvidia
Check whether the module is loaded:
lsmod | grep nvidia
Finally, run:
nvidia-smi
If the module fails to build, inspect the DKMS build log instead of repeatedly running the same installation command:
sudo find /var/lib/dkms/nvidia -name make.log
The log can reveal missing kernel headers, compiler errors or incompatibilities between the NVIDIA driver and the running kernel.
Note: DKMS is not used by every NVIDIA installation. Ubuntu also provides prebuilt, kernel-specific NVIDIA modules. If you use distribution-provided drivers, install the matching module package for the current kernel rather than assuming DKMS is available.
5. Fix DKMS Installation and Build Failures
Dynamic Kernel Module Support (DKMS) automatically builds kernel modules for installed Linux kernels when the required source code and development files are available. NVIDIA drivers can use DKMS to rebuild their kernel modules after kernel upgrades. However, the build process can fail if the required kernel headers, compiler tools or compatible driver sources are missing.
Step 1: Check DKMS status
Run:
sudo dkms status
A successful registration may display an entry similar to:
nvidia/XXX.XX, 6.x.x-xx-generic, x86_64: installed
The version numbers above are illustrative. Your actual output will depend on your driver and kernel versions. If the NVIDIA module is not listed, DKMS may not have been configured for that driver installation.
Step 2: Install the required build tools
For Ubuntu and Debian:
sudo apt update
sudo apt install build-essential dkms linux-headers-$(uname -r)
For Rocky Linux and AlmaLinux:
sudo dnf install gcc make dkms kernel-devel-$(uname -r) kernel-headers-$(uname -r)
Package availability and DKMS installation methods vary by distribution and repository configuration.
Step 3: Rebuild the module
Run:
sudo dkms autoinstall
Check the result:
sudo dkms status
If the module is successfully installed for the running kernel, load it:
sudo modprobe nvidia
Then verify:
nvidia-smi
What if DKMS still fails?
Check the relevant build log:
sudo cat /var/lib/dkms/nvidia/<driver-version>/build/make.log
Replace <driver-version> with the actual version shown by dkms status.
Look for errors relating to:
Missing kernel headers.
Unsupported kernel APIs.
GCC or compiler incompatibilities.
Failed compilation of NVIDIA source files.
Driver versions that do not support the installed kernel.
Do not assume that reinstalling DKMS will resolve every build failure. The underlying compilation error must be addressed first.
6. Fix Secure Boot Blocking the NVIDIA Driver
Secure Boot is a UEFI security feature that checks whether boot components and kernel modules meet the system's signature requirements. On Linux systems, Secure Boot can prevent an unsigned or untrusted NVIDIA kernel module from loading. This is particularly relevant when installing NVIDIA drivers through DKMS or third-party installation methods.
Step 1: Check Secure Boot status
On Ubuntu and Debian systems with mokutil installed, run:
mokutil --sb-state
If Secure Boot is enabled, the output will indicate that it is enabled.
Step 2: Check the module loading error
Run:
sudo modprobe nvidia
If the command returns an error such as Key was rejected by service or the kernel logs indicate that module signature verification failed, Secure Boot may be blocking the NVIDIA module.
Check the kernel logs:
sudo dmesg -T | grep -iE 'secure boot|signature|key|nvidia'
Step 3: Enrol a trusted signing key
If your DKMS installation provides a Machine Owner Key (MOK), you may need to enrol that key so that Secure Boot trusts the NVIDIA module. On supported Ubuntu installations, the process may use:
sudo mokutil --import /var/lib/shim-signed/mok/MOK.der
Follow the instructions to set a temporary enrolment password, then reboot the server. During the next boot, the MOK Manager screen should allow you to enrol the key. Confirm the enrolment and restart the system if prompted.
The exact key location depends on the driver installation method. Do not run the example command if that certificate does not exist on your system.
Alternative: Use distribution-provided signed drivers
On Ubuntu, using the recommended Ubuntu NVIDIA driver packages is often simpler because the distribution provides signed driver modules that support Secure Boot. For most production environments, keeping Secure Boot enabled and using trusted signed modules is preferable to disabling the security feature.
Important: Do not disable Secure Boot as the first troubleshooting step, especially on production servers. Resolve the module signing issue using the appropriate distribution or driver installation method.
7. Check for a Nouveau Driver Conflict
Nouveau is an open-source graphics driver included in many Linux distributions. It provides support for NVIDIA hardware without requiring the proprietary NVIDIA kernel driver. However, Nouveau and the proprietary NVIDIA driver can conflict when both attempt to manage the same GPU.
This may result in GPU detection problems or prevent the NVIDIA driver from loading correctly.
Step 1: Check whether Nouveau is loaded
Run:
lsmod | grep nouveau
If the command returns an entry containing nouveau, the module is currently loaded.
Step 2: Create a Nouveau blacklist configuration
If the proprietary NVIDIA driver is required and the logs confirm a Nouveau conflict, create a blacklist configuration:
echo "blacklist nouveau" | sudo tee /etc/modprobe.d/blacklist-nouveau.conf
echo "options nouveau modeset=0" | sudo tee -a /etc/modprobe.d/blacklist-nouveau.conf
Step 3: Rebuild the initramfs
On Ubuntu and Debian systems:
sudo update-initramfs -u
On Rocky Linux and AlmaLinux:
sudo dracut --force
Step 4: Reboot the server
sudo reboot
After the server starts again, check:
lsmod | grep nouveau
The command should return no output if Nouveau is no longer loaded. Then check the NVIDIA driver:
nvidia-smi
Important: Do not blacklist Nouveau simply because it is installed. First confirm that it is causing a conflict. This change can also affect graphical output, so make sure you have an appropriate recovery method before applying it to a remote production server.
8. Fix NVIDIA Driver and Library Version Mismatches
Another possible cause is a mismatch between the NVIDIA user-space libraries and the kernel module. For example, an NVIDIA driver update may install new user-space components without immediately replacing the older kernel module currently loaded in memory. This can happen after a driver package upgrade or a system maintenance operation.
A related error may look like this:
Failed to initialize NVML: Driver/library version mismatch
Step 1: Check the NVIDIA driver information
Run:
cat /proc/driver/nvidia/version
This displays information about the NVIDIA kernel driver currently loaded.
Step 2: Check the kernel logs
Run:
sudo dmesg -T | grep -i "NVRM"
Look for messages containing NVRM: API mismatch. These messages can indicate that the NVIDIA user-space driver and loaded kernel module have different versions.
Step 3: Restart the server
If the NVIDIA packages were upgraded successfully and the old module is still loaded, a reboot will normally load the updated kernel module.
sudo reboot
After restarting, verify:
nvidia-smi
Note: A reboot is appropriate when the mismatch is caused by an updated driver waiting for the next module load. If the driver packages themselves are inconsistent or the new module failed to build, fix the package or module installation before expecting a reboot to solve the issue.
9. Reinstall the NVIDIA Driver on Linux
If the previous troubleshooting steps do not resolve the issue, the NVIDIA driver installation may be incomplete, corrupted or incompatible with the installed GPU and kernel. Before reinstalling, identify your GPU model and the driver branch supported by that hardware.
NVIDIA driver installation methods differ between distributions. Avoid mixing distribution-managed packages with NVIDIA's standalone .run installer, as doing so can create conflicting files and make future updates more difficult.
Ubuntu and Debian
First, check the recommended driver:
ubuntu-drivers devices
This command is available on Ubuntu systems with the relevant driver management tools installed.
To install the recommended driver on Ubuntu:
sudo ubuntu-drivers install
Alternatively, you can install an appropriate driver package through APT, provided it is compatible with your GPU and distribution. On Debian, use the driver packages provided by the configured Debian repositories and follow the distribution's NVIDIA installation instructions.
After installation:
sudo reboot
Rocky Linux and AlmaLinux
On supported RHEL-based distributions, NVIDIA provides repository-based driver installation methods. The correct repository, module stream and driver package depend on the operating system version, GPU architecture and NVIDIA driver branch.
For example, NVIDIA's current open-kernel-module packaging uses commands such as:
sudo dnf module enable nvidia-driver:open-dkms
sudo dnf install nvidia-open
These commands apply to supported distributions and repository configurations; they are not universal installation commands for every Rocky Linux or AlmaLinux release. For older GPUs or unsupported open-kernel-module configurations, the appropriate proprietary driver package may be required instead.
After installing the correct driver:
sudo reboot
Verify the installation:
nvidia-smi
Driver compatibility matters: NVIDIA's open GPU kernel modules support Turing-generation GPUs and newer. Older Maxwell, Pascal and Volta GPUs require the compatible proprietary kernel module flavour. Always check NVIDIA's GPU and driver support documentation before selecting a driver package.
10. Verify That the NVIDIA GPU Driver Is Working
Once the driver has been repaired or reinstalled, verify that the Linux system can communicate with the GPU.
Run:
nvidia-smi
A successful response displays a GPU information table containing details such as:
GPU name and model.
Driver version.
CUDA compatibility information.
GPU temperature.
GPU utilisation.
GPU memory usage.
Power consumption.
Running GPU processes.
You can also run additional checks.
Check the NVIDIA driver version:
nvidia-smi --query-gpu=name,driver_version --format=csv
Check GPU utilisation:
nvidia-smi --query-gpu=utilization.gpu,memory.used,memory.total --format=csv
Check loaded NVIDIA modules:
lsmod | grep nvidia
Check GPU devices:
ls -l /dev/nvidia*
If nvidia-smi works but a CUDA application still cannot access the GPU, investigate the CUDA toolkit, application compatibility, container runtime or device permissions separately. A working NVIDIA driver does not automatically guarantee that every CUDA application is configured correctly.
11. Common NVIDIA-SMI Errors and Their Solutions
The following table summarises several related errors that Linux administrators may encounter when troubleshooting NVIDIA GPUs.
| Error or symptom | Possible cause | Recommended action |
|---|---|---|
| NVIDIA-SMI cannot communicate with the driver | NVIDIA kernel module is not loaded | Check lsmod, modinfo and kernel logs |
| Module nvidia not found | Module missing for the running kernel | Install the appropriate driver module or rebuild DKMS |
| Key was rejected by service | Secure Boot rejected the module signature | Enrol a trusted signing key or use signed drivers |
| Failed to initialise NVML: Driver/library version mismatch | Loaded module and user-space libraries have different versions | Check NVRM logs and restart after a successful driver update |
| No devices were found | GPU not accessible, driver conflict or hardware configuration issue | Check lspci, Nouveau status and PCIe passthrough |
| DKMS build failed | Missing headers or compilation incompatibility | Inspect the DKMS make.log and correct the build error |
| CUDA application cannot access GPU | CUDA, container or application configuration issue | Verify CUDA compatibility and NVIDIA container configuration |
12. How to Prevent NVIDIA-SMI Errors on Production Linux Servers
Fixing the driver is only part of maintaining a reliable GPU server. Regular maintenance and driver compatibility checks help prevent the same problem from happening again.
Consider the following practices:
Check driver compatibility before kernel upgrades: Before installing a new Linux kernel, confirm that the NVIDIA driver supports it and that the required prebuilt module or DKMS build process is available.
Maintain matching kernel headers: If your installation uses DKMS, ensure that the correct kernel headers and development tools are available whenever the kernel changes.
Avoid mixing NVIDIA installation methods: Use one consistent driver management method. Mixing APT, DNF and NVIDIA's standalone installer can create package conflicts.
Keep Secure Boot configuration consistent: Use trusted module signatures and verify that DKMS-generated modules can be loaded after driver updates.
Verify GPU functionality after maintenance: Run
nvidia-smiafter kernel upgrades, driver updates and server reboots. This helps identify problems before GPU-dependent applications are affected.Monitor GPU health: Regularly monitor GPU temperature, utilisation, memory usage and hardware-related kernel messages. Early detection can help identify driver and hardware problems before they cause service interruptions.
Frequently Asked Questions
1. Why does NVIDIA-SMI say it cannot communicate with the NVIDIA driver?
This error usually occurs because the NVIDIA kernel module is not loaded or cannot communicate with the GPU. Common causes include incomplete driver installation, kernel updates, DKMS build failures, Secure Boot restrictions and driver conflicts.
2. Can a Linux kernel update break the NVIDIA driver?
Yes. A kernel update can cause the NVIDIA driver to stop working if its kernel module is not available for the newly installed kernel. Installing the matching kernel headers and rebuilding the module, where applicable, can resolve the issue.
3. How do I check whether the NVIDIA driver is loaded on Linux?
Run lsmod | grep nvidia. If NVIDIA modules are listed, they are loaded into the running kernel. You can also use modinfo nvidia to inspect the installed module.
4. Does Secure Boot affect NVIDIA drivers?
Yes. Secure Boot can prevent unsigned or untrusted NVIDIA kernel modules from loading. Enrolling a trusted signing key or using distribution-provided signed driver packages can resolve this problem.
5. Do I need to reinstall CUDA to fix the NVIDIA-SMI error?
Not necessarily. nvidia-smi communicates with the NVIDIA driver through NVML. If the kernel driver is unavailable, reinstalling CUDA alone will not fix the underlying problem. Diagnose and repair the NVIDIA driver first.
6. How can I check whether Linux detects my NVIDIA GPU?
Run lspci | grep -i nvidia. If the GPU is not listed, investigate hardware detection, BIOS settings or GPU passthrough before reinstalling the driver.
Conclusion
The NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver error usually points to a problem between the NVIDIA user-space utilities and the Linux kernel driver. Kernel upgrades, missing kernel headers, DKMS build failures, Secure Boot restrictions and driver conflicts are important areas to investigate.
Start by checking whether Linux detects the GPU, inspect the loaded NVIDIA modules and review the kernel logs. Then apply the fix that matches the actual error rather than reinstalling the driver without identifying the cause.
For administrators managing AI workloads, machine learning environments and GPU-accelerated applications, maintaining compatible NVIDIA drivers and monitoring GPU health are essential parts of reliable Linux server management.
Explore GTZHost for infrastructure solutions designed to support demanding server workloads.