How to Install NVIDIA Drivers & CUDA on Ubuntu 26.04 for H100 GPUs

Setting up an NVIDIA H100 on Ubuntu 26.04 LTS ("Resolute Raccoon") is more straightforward than it used to be — Ubuntu 26.04 ships CUDA and the NVIDIA driver stack directly from its own repositories. But data center GPUs like the H100 have a few extra requirements that consumer cards don't, including open-source kernel modules and, on multi-GPU NVLink systems, a fabric manager service. This guide covers the full setup from a clean Ubuntu 26.04 install through to a verified, working CUDA environment.

If you're setting this up on rented hardware, this process applies whether you're on a personal workstation or a rack-mounted GPU dedicated server in a data center.

Before You Start: What You'll Need

• A fresh install of Ubuntu 26.04 LTS, x86_64, with sudo access

• An NVIDIA H100 (PCIe or SXM variant) installed and recognized on the PCIe bus

• At least 8 GB of free disk space for the driver, toolkit, and sample programs

• A stable internet connection to pull packages from Ubuntu's archive and NVIDIA's repository

• BIOS setting "Above 4G Decoding" / "Large BARs" enabled — required for the H100 to be addressed correctly by the OS

If you're deploying on a hosted server rather than local hardware, check your provider's GPU servers or GPU dedicated servers – USA page for available H100 configurations, or reach out through our contact page if you need a custom build.

Step 1: Update the System and Install Build Tools

Start with a clean, updated base and the packages needed to build kernel modules:

sudo apt update && sudo apt upgrade -y
sudo apt install -y build-essential linux-headers-$(uname -r) dkms wget curl

dkms is important here — it rebuilds the NVIDIA kernel module automatically whenever the kernel updates, so a routine apt upgrade doesn't break your GPU driver.

Step 2: Blacklist the Nouveau Driver

Ubuntu ships with the open-source Nouveau driver by default, and it will conflict with NVIDIA's driver if left active. Blacklist it:

sudo tee /etc/modprobe.d/blacklist-nouveau.conf << EOF
blacklist nouveau
options nouveau modeset=0
EOF

sudo update-initramfs -u
sudo reboot

This is the only reboot in the whole process, and it's needed to unload Nouveau before the NVIDIA driver can load in its place.

Step 3: Install the NVIDIA Open GPU Kernel Modules

For Hopper-generation cards like the H100, NVIDIA's open-source kernel modules are the recommended path, not the legacy proprietary ones. Add NVIDIA's CUDA repository and keyring:

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2604/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt update

Then install the open kernel module driver along with the utilities:

sudo apt install -y nvidia-driver-open nvidia-utils

If that package name isn't available in your current repo snapshot, use the standard metapackage instead — Ubuntu 26.04's restricted repository already tracks the current production driver branch:

sudo apt install -y nvidia-driver

Reboot is not required for this step in most cases, since Nouveau was already removed in Step 2 and the module loads directly.

Step 4: Install the CUDA Toolkit

With the repository already added, install the full CUDA toolkit:

sudo apt install -y cuda-toolkit

Add CUDA's binaries to your shell path so nvcc and other tools are available:

echo 'export PATH=/usr/local/cuda/bin:$PATH' >> ~/.bashrc
echo 'export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH' >> ~/.bashrc
source ~/.bashrc

Step 5: Verify the Installation

Confirm the driver sees your H100:

nvidia-smi

You should see the GPU listed as NVIDIA H100, along with driver version, CUDA version, and current memory usage. Then confirm the toolkit itself:

nvcc --version

If both commands return clean output with no errors, your driver and toolkit are correctly installed and matched.

Step 6: Multi-GPU Setup — Install NVIDIA Fabric Manager

If your server has multiple H100s connected over NVLink (common on SXM-based systems), you need the fabric manager service so the GPUs can communicate directly instead of routing through the CPU:

sudo apt install -y nvidia-fabricmanager-<driver-version>
sudo systemctl enable nvidia-fabricmanager
sudo systemctl start nvidia-fabricmanager

Replace <driver-version> with the major version reported by nvidia-smi. Without this service running, multi-GPU jobs using NVLink will fall back to slower PCIe communication or fail to initialize the topology correctly.

Step 7: Enable Persistence Mode

On a dedicated GPU server, you generally want the driver to stay initialized between jobs instead of unloading when idle, which avoids delays each time a new process starts:

sudo nvidia-smi -pm 1

To make this persist across reboots, create a systemd service or add it to your server's startup scripts.

Troubleshooting Common Issues

• nvidia-smi returns "No devices were found": Usually means Nouveau is still loaded, or the BIOS setting for Above 4G Decoding is disabled. Re-check Step 2 and your BIOS settings.

• Driver installs but CUDA version mismatch errors appear: This happens when the driver and toolkit come from different repositories. Stick to one source (NVIDIA's CUDA repo, as used in this guide) for both, rather than mixing Ubuntu's default driver with a toolkit downloaded separately from NVIDIA's website.

• nvcc: command not found after installation: Your shell path wasn't updated. Re-run Step 4's export PATH commands and confirm they were added to ~/.bashrc, not just run once in the current session.

• Multi-GPU jobs are slower than expected: Check that nvidia-fabricmanager is actually running (systemctl status nvidia-fabricmanager) — if it's not active, NVLink isn't being used.

Next Steps

Looking for H100 capacity without managing the hardware yourself? Browse GTZHost's GPU servers or check the GPU dedicated servers – USA page, or get in touch through our contact us page for a custom configuration hosted in one of our data centers.