AI server Build

Ubunut 24.04

[[:tech:Linux:AI Setup]] \\

====== Install all drivers needed for using V100 on Ubuntu 24.04 LTS ======

Setting up an enterprise data center GPU like the NVIDIA V100 (Volta architecture) on **Ubuntu 24.04 LTS** requires the proprietary driver and the CUDA Toolkit. Since the V100 is typically used in a headless server environment for compute, machine learning, and AI workloads, a **compute-only/headless driver installation** is ideal. This avoids dragging in unnecessary graphical desktop packages.

Here is the clean, streamlined method using the official NVIDIA network repositories for Ubuntu 24.04.

==== Step 1: Clean Up Existing Drivers ====

To prevent conflicts with generic open-source or older drivers, remove any current installations:

<code bash\>
sudo apt-get purge 'nvidia\*' 'cuda\*' -y  
sudo apt-get autoremove -y  
</code>

==== Step 2: Install Linux Kernel Headers ====

Ensure you have the correct kernel headers installed so the NVIDIA kernel modules can compile properly:

<code bash\>
sudo apt-get update  
sudo apt-get install linux-headers-$(uname -r) build-essential -y  
</code>

==== Step 3: Setup the NVIDIA & CUDA Network Repository ====

NVIDIA provides a repository pinned to Ubuntu 24.04 (noble). Fetch the repository configurations and GPG keys so apt can safely pull the latest stable packages:

<code bash\>

# Download the repository pin file to set priority

wget <https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86>\_64/cuda-ubuntu2404.pin  
sudo mv cuda-ubuntu2404.pin /etc/apt/preferences.d/cuda-repository-pin-600

# Download and install the repository package

wget <https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86>\_64/cuda-keyring\_1.1-1\_all.deb  
sudo dpkg -i cuda-keyring\_1.1-1\_all.deb

# Update apt package listings

sudo apt-get update  
</code>

==== Step 4: Install the Drivers and CUDA Toolkit ====

For enterprise cards like the V100, the stable branch (such as the nvidia-driver-550-server or nvidia-driver-565-server depending on repository syncs) is highly recommended.

To fetch the **headless, data-center optimal drivers** alongside the CUDA toolkit, run:

<code bash\>
sudo apt-get -y install cuda-toolkit-12-8 nvidia-headless-server-550 nvidia-utils-550  
</code>

*Note: If you plan on deploying Docker containers that utilize the V100, you should install the container runtime package as well:* `sudo apt-get install -y nvidia-container-toolkit`

==== Step 5: Update Environment Variables ====

To ensure your system paths find the CUDA binaries, append them to your shell configuration (\~/.bashrc):

<code bash\>
echo 'export PATH=/usr/local/cuda/bin${PATH:+:${PATH}}' \>\> \~/.bashrc  
echo 'export LD\_LIBRARY\_PATH=/usr/local/cuda/lib64${LD\_LIBRARY\_PATH:+:${LD\_LIBRARY\_PATH}}' \>\> \~/.bashrc  
source \~/.bashrc  
</code>

==== Step 6: Reboot and Verify ====

Restart the machine to initialize the newly compiled kernel modules:

<code bash\>
sudo reboot  
</code>

Once the system is back up, verify that the OS recognizes the V100 and that the drivers are operational:

<code bash\>
nvidia-smi  
</code>

You should see an output matrix detailing your V100, its current temperature, power consumption, and the driver/CUDA version currently running.


====== GeForce 210 Driver Override Guide ======
We cannot fix this the ''nouveau'' driver cannot live with the V100 NVidia driver


====== Installing Docker on Ubuntu 24.04 ======

To install **Docker Engine** on **Ubuntu 24.04**, the official and recommended method is to set up Docker's official ''apt'' repository. This ensures you get the latest stable version and automatic future updates.

===== Step 1: Remove Conflicting Packages =====

Before starting, remove any default or unofficial Docker packages to prevent conflicts:

<code>
sudo apt remove docker.io docker-doc docker-compose docker-compose-v2 podman-docker containerd runc
</code>

===== Step 2: Set Up the Docker Repository =====

Update your package index and install the prerequisite security certificates:

<code>
sudo apt update
sudo apt install -y ca-certificates curl gnupg
</code>

Next, add Docker's official GPG key to verify package authenticity:

<code>
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://docker.com -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc
</code>

Add the stable repository to your system's ''apt'' sources:

<code>
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://docker.com \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
</code>

===== Step 3: Install Docker Engine & Plugins =====

Update your repository index to include Docker's packages, then install Docker Engine, the CLI, and **Docker Compose V2**:

<code>
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
</code>

===== Step 4: Verify the Installation =====

Check if the service is up and running:

<code>
sudo systemctl status docker
</code>

Run the default test container to verify it pulls and executes properly:

<code>
sudo docker run hello-world
</code>

----

===== Optional: Run Docker Without Sudo =====

By default, Docker requires root privileges. To run commands as a regular user, add yourself to the ''docker'' group:

  * Create the group (usually exists already):
<code>
sudo groupadd docker
</code>
  * Add your user to the group:
<code>
sudo usermod -aG docker $USER
</code>
  * Apply the changes immediately without logging out:
<code>
newgrp docker
</code>
  * Verify by running without ''sudo'':
<code>
docker run hello-world
</code>

For detailed troubleshooting or specific alternative guides, you can refer to the [[https://docker.com|Official Docker Engine Installation Docs]].


====== Enabling NVIDIA AI and GPU Acceleration on Docker ======

To enable NVIDIA AI and GPU acceleration on Docker, you must install and configure the **NVIDIA Container Toolkit**. This acts as the bridge that allows Docker containers to access your host machine's physical GPU.

Here is how to set it up on a Linux host (such as Ubuntu).

===== 1. Install Prerequisites =====

Ensure you have the official NVIDIA drivers installed on your host system, along with Docker Engine. Validate your driver installation by running:

<code>
nvidia-smi
</code>

//(You should see a table displaying your GPU information.)//

===== 2. Install the NVIDIA Container Toolkit =====
https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html

Add the official package repositories and install the toolkit:

<code>
# Add the NVIDIA package repository
curl -fsSL https://github.io | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo sed -i -e '/experimental/ s/^#//g' /etc/apt/sources.list.d/nvidia-container-toolkit.list

# Update package list and install
sudo apt-get update

export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.19.1-1
  sudo apt-get install -y \
      nvidia-container-toolkit=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
      nvidia-container-toolkit-base=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
      libnvidia-container-tools=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
      libnvidia-container1=${NVIDIA_CONTAINER_TOOLKIT_VERSION}

</code>

===== 3. Configure the Docker Runtime =====

Configure the Docker daemon to automatically recognize the NVIDIA container runtime, then restart the Docker service:

<code>
# Configure Docker to use the NVIDIA runtime
sudo nvidia-ctk runtime configure --runtime=docker

# Restart the Docker service to apply changes
sudo systemctl restart docker
</code>

===== 4. Verify GPU Access in Docker =====

Test the configuration by running a lightweight, official CUDA container. Passing the ''--gpus all'' flag tells Docker to expose your graphics cards to the container:

<code>
docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi
</code>

If successful, the container will print out your host GPU details exactly like the native ''nvidia-smi'' command did. 

----

===== Alternative: Docker Desktop (Windows / Mac) =====

If you are developing locally via Docker Desktop, you do not need to manually install the toolkit. Instead, you can leverage native AI features:
  * **WSL 2 (Windows)**: Ensure WSL integration is turned on under **Settings > Resources > WSL Integration** in Docker Desktop. 
  * **Docker Model Runner (DMR)**: Go to **Settings > AI**, check **Enable Docker Model Runner**, and select **Enable GPU-backed inference** to pull and manage local AI models natively using commands like ''docker model run''.

If you plan to deploy enterprise-grade AI workloads, you can also authenticate your Docker client to the NVIDIA NGC Container Registry using an API key. This gives you direct access to fine-tuned, GPU-optimized AI models and microservices.


====== Installing LM Studio on Ubuntu 24.04 ======

To install LM Studio on Ubuntu 24.04 LTS, you have two primary options: the **headless lms daemon** (ideal for servers, command-line usage, or API orchestration) or the **AppImage** (for the full graphical interface).

---

===== Option 1: Headless Daemon (lms) =====
If you intend to use LM Studio for backend services, API hosting, or CI/CD, the ''lms'' daemon is the official, recommended approach.

==== 1. Install the Daemon ====
Run the following command in your terminal to download and execute the official installer:
<code>
curl -fsSL https://lmstudio.ai/install.sh | bash
</code>

==== 2. Usage ====
Once the installation script completes, you can interact with the service directly using the ''lms'' CLI.

To start the background service:
<code>
lms daemon up
</code>

**Note:** If you encounter an "Illegal instruction" error upon execution, ensure your system's ''libc'' and environment variables are fully updated. This can sometimes occur on bleeding-edge hardware configurations.

---

===== Option 2: Graphical AppImage (Full GUI) =====
If you require the standard LM Studio desktop interface, you must download the AppImage. Note that Ubuntu 24.04 enforces strict sandboxing rules that require manual adjustment.

==== 1. Download and Make Executable ====
Download the latest Linux AppImage from the official LM Studio website, then give it execution permissions:
<code>
chmod +x LM_Studio-*.AppImage
</code>

==== 2. Extract and Configure Sandbox ====
The application will likely fail to launch unless you fix the ''chrome-sandbox'' permissions. Extract the AppImage and update the owner permissions:
<code>
# Extract the AppImage
./LM_Studio-*.AppImage --appimage-extract

# Navigate to the extracted folder
cd squashfs-root

# Fix sandbox permissions
sudo chown root:root chrome-sandbox
sudo chmod 4755 chrome-sandbox
</code>

==== 3. Launch the Application ====
You can now run the application from within the extracted directory:
<code>
./lm-studio
</code>

If you still encounter sandboxing issues and prefer to bypass it, you can launch with the ''--no-sandbox'' flag (use with caution):
<code>
./lm-studio --no-sandbox
</code>

---

===== Troubleshooting & Performance =====
^ Issue ^ Resolution ^
| **Missing Dependencies** | Install base libraries: \\ <code>sudo apt install libatk1.0-0 libatk-bridge2.0-0 libcups2 libgdk-pixbuf2.0-0 libgtk-3-0 libpango-1.0-0 libcairo2 libxcomposite1 libxdamage1 libasound2t64 libatspi2.0-0</code> |
| **Hardware Acceleration** | For integrated or discrete GPUs, ensure your drivers (such as ''intel-media-va-driver-non-free'' for Intel setups) are properly configured via ''mesa'' to ensure local model inference acceleration. |