How to Install NVIDIA Container Toolkit with Docker on Ubuntu 26.04

If you're running AI or machine learning workloads in containers, sooner or later you hit the same wall: Docker doesn't know your GPU exists. A container built from nvidia/cuda will start fine, but the moment it tries to touch the GPU, it fails — because by default, Docker has no way to pass through NVIDIA hardware to an isolated container environment.

The NVIDIA Container Toolkit solves this. It's the official bridge between the NVIDIA driver on your host and the Docker runtime, letting containers use the GPU without bundling drivers inside the image itself. This guide walks through installing it on Ubuntu 26.04, from checking your GPU to running a verified CUDA container.

Who This Guide Is For

This tutorial assumes you're setting up a bare-metal or VPS server with an NVIDIA GPU (RTX, A100, H100, L40S, etc.) for AI/ML, video processing, or general CUDA workloads, and you already have basic comfort with the Linux command line. If you're provisioning a fresh dedicated GPU server, this is typically one of the first things you'll configure after the OS install.

Prerequisites

Before starting, confirm the following:

  • Ubuntu 26.04 installed (server or desktop edition)

  • Root or sudo access

  • An NVIDIA GPU physically installed and recognized by the system

  • Docker Engine installed (we'll cover this below if it isn't yet)

  • A stable internet connection to pull packages and container images

Step 1: Confirm Your GPU Is Detected

Before touching drivers or Docker, confirm the system actually sees the GPU at the hardware level:

bash 
lspci | grep -i nvidia

You should see output listing your GPU model (for example, an RTX 4090 or an H100 SXM). If nothing returns, stop here — this is a hardware or BIOS/passthrough issue, not a software one, and installing the toolkit won't help until the card is visible to the OS.

Step 2: Install the NVIDIA Driver

The Container Toolkit does not replace the NVIDIA driver — it depends on it. The driver has to be installed directly on the host, not inside a container.

The simplest approach is to let Ubuntu detect and install the recommended driver automatically:

bash 
sudo ubuntu-drivers install

If you need a specific driver version — for example, to match a particular CUDA toolkit version required by your workload — install it explicitly instead:

bash 
sudo apt update
sudo apt install -y nvidia-driver-550

Reboot after the driver installs, since the kernel module won't load into a running session:

bash 
sudo reboot

Once the server is back up, verify the driver loaded correctly:

bash 
nvidia-smi

This should print a table with your GPU name, driver version, and current CUDA version supported by that driver. If this command fails, the container toolkit will fail too — resolve the driver first.

Step 3: Install Docker Engine

If Docker isn't installed yet, add Docker's official APT repository rather than using the outdated docker.io package from Ubuntu's default repos:

bash 
sudo apt update
sudo apt install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo $VERSION_CODENAME) stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Confirm the install:

bash 
sudo docker run hello-world

If this prints a welcome message, Docker is working correctly and ready for GPU configuration.

Step 4: Add the NVIDIA Container Toolkit Repository

NVIDIA maintains its own APT repository for the container toolkit, separate from the driver. Add the GPG key and repo list:

bash 
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
  sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

Update the package index so APT picks up the new source:

bash 
sudo apt update

Step 5: Install the NVIDIA Container Toolkit

With the repository in place, installation is a single command:

bash 
sudo apt install -y nvidia-container-toolkit

This pulls in nvidia-container-toolkit-base, libnvidia-container1, and the CLI utilities needed to hook the toolkit into Docker's runtime.

Step 6: Configure the Docker Runtime

The toolkit ships with a helper command that edits Docker's daemon configuration automatically, rather than hand-editing /etc/docker/daemon.json yourself:

bash 
sudo nvidia-ctk runtime configure --runtime=docker

This registers the nvidia runtime inside Docker's daemon config. Restart Docker to apply the change:

bash 
sudo systemctl restart docker

Step 7: Verify GPU Access Inside a Container

This is the step that actually confirms everything is wired together correctly. Run a CUDA container and check if it can see the GPU:

bash 
docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi

If configured correctly, you'll see the same nvidia-smi output as on the host — GPU name, memory usage, driver version — but this time it's running from inside an isolated container. That confirms the container runtime is correctly passing GPU access through.

Common Errors and How to Fix Them

  • docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]] This means Docker doesn't know about the nvidia runtime yet. Re-run Step 6 and make sure systemctl restart docker actually completed without errors.

  • nvidia-smi: command not found inside the container This usually means the base image doesn't include the NVIDIA userspace libraries. Use an nvidia/cuda base image rather than plain ubuntu, since the toolkit injects driver libraries into CUDA-aware images at runtime.

  • CUDA version mismatch errors The CUDA version inside your container must be equal to or older than the driver version installed on the host — never newer. Check your host driver's supported CUDA version with nvidia-smi | grep "CUDA Version", and pick a container image at or below that number.

  • GPU not visible even after correct setup Double check /dev/nvidia* device files exist on the host with ls -la /dev/nvidia*. If they're missing, the driver may not have loaded correctly after reboot — recheck Step 2.

Next Steps

With GPU-accelerated Docker containers working, a natural next step is setting up a full inference or training environment on top of this base — see our guide on setting up vLLM for multi-GPU inference, or if you're deploying a bare-metal AI server from scratch, check out deploying and optimizing an NVIDIA GPU server for AI.

This guide was tested on Ubuntu 26.04 LTS with Docker Engine and NVIDIA Container Toolkit. Commands may need adjustment for non-LTS releases or older driver branches. Last updated August 2026.