Setting up an enterprise data center GPU like the NVIDIA V100 (Volta architecture) on Ubuntu 24.04 LTS requires the proprietary driver and the CUDA Toolkit. Since the V100 is typically used in a headless server environment for compute, machine learning, and AI workloads, a compute-only/headless driver installation is ideal. This avoids dragging in unnecessary graphical desktop packages.

Here is the clean, streamlined method using the official NVIDIA network repositories for Ubuntu 24.04.

To prevent conflicts with generic open-source or older drivers, remove any current installations:

sudo apt-get purge 'nvidia\*' 'cuda\*' -y  
sudo apt-get autoremove -y  

Ensure you have the correct kernel headers installed so the NVIDIA kernel modules can compile properly:

sudo apt-get update  
sudo apt-get install linux-headers-$(uname -r) build-essential -y  

NVIDIA provides a repository pinned to Ubuntu 24.04 (noble). Fetch the repository configurations and GPG keys so apt can safely pull the latest stable packages:

# Download the repository pin file to set priority
 
wget <https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86>\_64/cuda-ubuntu2404.pin  
sudo mv cuda-ubuntu2404.pin /etc/apt/preferences.d/cuda-repository-pin-600
 
# Download and install the repository package
 
wget <https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86>\_64/cuda-keyring\_1.1-1\_all.deb  
sudo dpkg -i cuda-keyring\_1.1-1\_all.deb
 
# Update apt package listings
 
sudo apt-get update  

For enterprise cards like the V100, the stable branch (such as the nvidia-driver-550-server or nvidia-driver-565-server depending on repository syncs) is highly recommended.

To fetch the headless, data-center optimal drivers alongside the CUDA toolkit, run:

sudo apt-get -y install cuda-toolkit-12-8 nvidia-headless-server-550 nvidia-utils-550  

*Note: If you plan on deploying Docker containers that utilize the V100, you should install the container runtime package as well:* `sudo apt-get install -y nvidia-container-toolkit`

To ensure your system paths find the CUDA binaries, append them to your shell configuration (\~/.bashrc):

echo 'export PATH=/usr/local/cuda/bin${PATH:+:${PATH}}' \>\> \~/.bashrc  
echo 'export LD\_LIBRARY\_PATH=/usr/local/cuda/lib64${LD\_LIBRARY\_PATH:+:${LD\_LIBRARY\_PATH}}' \>\> \~/.bashrc  
source \~/.bashrc  

Restart the machine to initialize the newly compiled kernel modules:

sudo reboot  

Once the system is back up, verify that the OS recognizes the V100 and that the drivers are operational:

nvidia-smi  

You should see an output matrix detailing your V100, its current temperature, power consumption, and the driver/CUDA version currently running.

We cannot fix this the nouveau driver cannot live with the V100 NVidia driver

To install Docker Engine on Ubuntu 24.04, the official and recommended method is to set up Docker's official apt repository. This ensures you get the latest stable version and automatic future updates.

Before starting, remove any default or unofficial Docker packages to prevent conflicts:

sudo apt remove docker.io docker-doc docker-compose docker-compose-v2 podman-docker containerd runc

Update your package index and install the prerequisite security certificates:

sudo apt update
sudo apt install -y ca-certificates curl gnupg

Next, add Docker's official GPG key to verify package authenticity:

sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://docker.com -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

Add the stable repository to your system's apt sources:

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://docker.com \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

Update your repository index to include Docker's packages, then install Docker Engine, the CLI, and Docker Compose V2:

sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Check if the service is up and running:

sudo systemctl status docker

Run the default test container to verify it pulls and executes properly:

sudo docker run hello-world

By default, Docker requires root privileges. To run commands as a regular user, add yourself to the docker group:

• Create the group (usually exists already):

sudo groupadd docker

• Add your user to the group:

sudo usermod -aG docker $USER

• Apply the changes immediately without logging out:

newgrp docker

• Verify by running without sudo:

docker run hello-world

For detailed troubleshooting or specific alternative guides, you can refer to the Official Docker Engine Installation Docs.

To enable NVIDIA AI and GPU acceleration on Docker, you must install and configure the NVIDIA Container Toolkit. This acts as the bridge that allows Docker containers to access your host machine's physical GPU.

Here is how to set it up on a Linux host (such as Ubuntu).

Ensure you have the official NVIDIA drivers installed on your host system, along with Docker Engine. Validate your driver installation by running:

nvidia-smi

(You should see a table displaying your GPU information.)

https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html

Add the official package repositories and install the toolkit:

# Add the NVIDIA package repository
curl -fsSL https://github.io | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo sed -i -e '/experimental/ s/^#//g' /etc/apt/sources.list.d/nvidia-container-toolkit.list

# Update package list and install
sudo apt-get update

export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.19.1-1
  sudo apt-get install -y \
      nvidia-container-toolkit=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
      nvidia-container-toolkit-base=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
      libnvidia-container-tools=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
      libnvidia-container1=${NVIDIA_CONTAINER_TOOLKIT_VERSION}

Configure the Docker daemon to automatically recognize the NVIDIA container runtime, then restart the Docker service:

# Configure Docker to use the NVIDIA runtime
sudo nvidia-ctk runtime configure --runtime=docker

# Restart the Docker service to apply changes
sudo systemctl restart docker

Test the configuration by running a lightweight, official CUDA container. Passing the –gpus all flag tells Docker to expose your graphics cards to the container:

docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

If successful, the container will print out your host GPU details exactly like the native nvidia-smi command did.

If you are developing locally via Docker Desktop, you do not need to manually install the toolkit. Instead, you can leverage native AI features:

• WSL 2 (Windows): Ensure WSL integration is turned on under Settings > Resources > WSL Integration in Docker Desktop.
• Docker Model Runner (DMR): Go to Settings > AI, check Enable Docker Model Runner, and select Enable GPU-backed inference to pull and manage local AI models natively using commands like docker model run.

If you plan to deploy enterprise-grade AI workloads, you can also authenticate your Docker client to the NVIDIA NGC Container Registry using an API key. This gives you direct access to fine-tuned, GPU-optimized AI models and microservices.