====== GPU Monitoring Setup (Dell Precision 7180) ======

This guide explains how to monitor the usage, temperature, and metrics of two NVIDIA V100 GPUs on an Ubuntu 24.04 workstation using a professional-grade monitoring stack.

====== The Stack ======

The industry standard for high-end GPU monitoring is the **Prometheus + Grafana + NVIDIA DCGM Exporter** stack:

*   **NVIDIA DCGM Exporter:** Scrapes GPU metrics (Temperature, Power, Memory, Clock Speeds) and converts them into a format databases can read.
*   **Prometheus:** A time-series database that collects and stores the metrics every few seconds.
*   **Grafana:** The visualization layer that turns raw numbers into interactive, professional-grade graphs.

====== Docker Compose Implementation (Recommended) ======

This is the cleanest method for Ubuntu 24.04. It keeps all monitoring tools isolated in containers.

==== Prerequisites: NVIDIA Container Toolkit ====
To allow Docker to "see" your V100 GPUs, you must install the toolkit:
{{
# Configure the repository
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

# Update and install
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

# Configure Docker to use the NVIDIA driver
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
}}

==== Step-by-Step Setup ====

1. **Create a Directory:**
    {{
    mkdir ~/gpu-monitor
    cd ~/gpu-monitor
    }}

2. **Create Prometheus Configuration:**
    Create `prometheus.yml`:
    {{
    global:
      scrape_interval: 5s

    scrape_configs:
      - job_name: 'nvidia-gpu'
        static_configs:
          - targets: ['nvidia-dcgm-exporter:9400']
    }}

3. **Create Docker Compose File:**
    Create `docker-compose.yml`:
    {{
    version: '3.8'

    services:
      nvidia-dcgm-exporter:
        image: nvidia/dcgm-exporter:latest
        container_name: nvidia-dcgm-exporter
        deploy:
          resources:
            reservations:
              devices:
                - driver: nvidia
                  count: all
                  capabilities: [gpu]
        ports:
          - "9400:9400"
        restart: always

      prometheus:
        image: prom/prometheus:latest
        container_name: prometheus
        volumes:
          - ./prometheus.yml:/etc/prometheus/prometheus.yml
          - prometheus_data:/prometheus
        ports:
          - "9090:9090"
        restart: always

      grafana:
        image: grafana/grafana:latest
        container_name: grafana
        ports:
          - "3000:3000"
        volumes:
          - grafana_data:/var/lib/grafana
        restart: always
        environment:
          - GF_SECURITY_ADMIN_PASSWORD=admin

    volumes:
      prometheus_data:
      grafana_data:
    }}

4. **Launch the Stack:**
    {{
    docker compose up -d
    }}

====== Verification & Troubleshooting ======

*   **Checking the Exporter:**
    Check if the container is actually seeing the hardware by running:
    {{
    docker logs nvidia-dcgm-exporter
    }}
    *Look for: "DCGM successfully initialized!"*

*   **Verifying the Data Feed:**
    Visit `http://<your-ip>:9400` in your browser. You should see a wall of raw text (metrics).

*   **Connecting Grafana to Prometheus:**
    1. Open Grafana (`http://<your-ip>:3000`).
    2. Go to **Connections** -> **Data Sources** -> **Add Prometheus**.
    3. Set the URL to: `http://prometheus:9090` (Use the Docker service name, not localhost).
    4. Click **Save & Test**.

====== Finding Dashboards ======

To get the actual visual graphs:
1. Visit the [Grafana Dashboard Marketplace](https://grafana.com/grafana/dashboards/).
2. Search for **"NVIDIA DCGM"**.
3. Find a dashboard you like (e.g., ID **17499**).
4. In your Grafana instance, click **Import** and enter the ID, or download the **JSON** file from the website and upload it via the Import menu.