GPU Scheduling in Kubernetes: Device Plugins, GPU Operator & MIG

Kubernetes knows how to schedule CPU and memory. It has no built-in concept of a GPU.

A standard node description looks like this:

cpu: 64
memory: 256Gi

The 8 H100 GPUs on that same node are completely invisible to the scheduler — unless something tells it they exist.

This post covers the full stack that makes GPU scheduling work: how GPUs become visible, how they get inside containers, how the GPU Operator automates the whole setup, and how MIG and time-slicing let you share expensive hardware efficiently.

---

The Device Plugin Framework

Kubernetes uses the Device Plugin Framework to support hardware accelerators — GPUs, FPGAs, network adapters, or any other specialized device.

A Device Plugin is a gRPC server that runs as a DaemonSet on every node that has the hardware. It registers with the Kubelet and handles two operations:

ListAndWatch — Advertising Devices

The plugin calls ListAndWatch to stream the list of available devices to Kubelet continuously. Kubelet then updates the node's capacity in the API Server.

sequenceDiagram

participant Plugin as Device Plugin <br/> (DaemonSet)
participant Kubelet
participant API as API Server

Plugin->>Kubelet: Register (resourceName: nvidia.com/gpu)
Kubelet-->>Plugin: OK
Plugin->>Kubelet: ListAndWatch → [GPU-0, GPU-1, ..., GPU-7] (Healthy)
Kubelet->>API: Update Node capacity <br/> nvidia.com/gpu: 8

After this, kubectl describe node shows:

Capacity:
  nvidia.com/gpu:  8
Allocatable:
  nvidia.com/gpu:  8

Allocate — Assigning a Device to a Container

When the Kubelet starts a container that requested a GPU, it calls Allocate on the Device Plugin, which returns the environment variables and device mounts needed.

sequenceDiagram

participant Kubelet
participant Plugin as Device Plugin
participant Runtime as Container Runtime

Kubelet->>Plugin: Allocate(deviceIDs: [GPU-3])
Plugin-->>Kubelet: AllocateResponse <br/> (envs: CUDA_VISIBLE_DEVICES=3 <br/>  devices: /dev/nvidia3, /dev/nvidiactl)
Kubelet->>Runtime: Start container with <br/> env + device mounts

The container sees only its assigned GPU. Other GPUs on the node are invisible inside the container.

Requesting GPUs in a Pod

resources:
  limits:
    nvidia.com/gpu: "1"   # request exactly 1 GPU

Two rules that differ from CPU and memory:

1. No requests field — only limits. You cannot request 0.5 of a GPU.
2. Non-overcommittable — the scheduler never assigns more GPUs than physically available. There is no concept of GPU bursting.

For multi-GPU pods:

resources:
  limits:
    nvidia.com/gpu: "8"   # request all 8 GPUs on a DGX node

---

nvidia-container-toolkit

The Device Plugin tells Kubernetes which GPU to assign. But how does a GPU actually get inside a container?

That is the job of nvidia-container-toolkit (formerly nvidia-docker).

It is a container runtime hook that intercepts container creation and injects GPU access:

flowchart TD

Kubelet-->ContainerD["containerd"] -->
NVHook["nvidia-container-hook <br/> (OCI hook)"] -->
Inject["/dev/nvidia3 <br/> /dev/nvidiactl <br/> /dev/nvidia-uvm <br/> CUDA libraries <br/> CUDA_VISIBLE_DEVICES=3"] -->
Container["Container <br/> (sees GPU 3 only)"]

What gets injected:

Item	Purpose
/dev/nvidia3	The actual GPU device file
/dev/nvidiactl	NVIDIA control device
/dev/nvidia-uvm	Unified Memory driver
CUDA shared libraries	Mounted from host into container
CUDA_VISIBLE_DEVICES=3	Tells CUDA which GPU index is visible

This is why GPU containers work without bundling CUDA drivers — they use the host's drivers via device file injection.

---

NVIDIA GPU Operator

Installing GPU drivers, container toolkit, device plugin, and monitoring agents across hundreds of nodes manually is error-prone and hard to keep consistent.

The GPU Operator is a Kubernetes Operator that manages the entire GPU software stack as a single reconciliation loop.

flowchart LR

GPUOperator["GPU Operator <br/> (reconciles ClusterPolicy)"]

GPUOperator -->Driver["NVIDIA Driver <br/> (DaemonSet — compiles kernel module)"]
GPUOperator-->Toolkit["nvidia-container-toolkit <br/> (DaemonSet — injects GPU into containers)"]
GPUOperator-->DevicePlugin["Device Plugin <br/> (DaemonSet — ListAndWatch + Allocate)"]
GPUOperator-->DCGM["DCGM Exporter <br/> (DaemonSet — Prometheus metrics)"]
GPUOperator-->MIGManager["MIG Manager <br/> (DaemonSet — configures MIG partitions)"]
GPUOperator-->NodeFeatureDiscovery["Node Feature Discovery <br/> (labels GPU nodes)"]

ClusterPolicy

Everything the GPU Operator manages is declared in a single CRD: ClusterPolicy.

apiVersion: nvidia.com/v1
kind: ClusterPolicy
metadata:
  name: gpu-cluster-policy
spec:
  driver:
    enabled: true
    version: "535.104.12"
    repository: nvcr.io/nvidia

  toolkit:
    enabled: true
    version: "v1.14.3-centos7"

  devicePlugin:
    enabled: true
    version: "v0.14.5"
    config:
      name: device-plugin-config   # ConfigMap with sharing config

  dcgmExporter:
    enabled: true
    version: "3.3.0-3.2.0-ubuntu22.04"
    config:
      name: dcgm-metrics-config

  mig:
    strategy: single   # single: all GPUs use same MIG profile
                       # mixed: GPUs can have different profiles

  migManager:
    enabled: true

The GPU Operator watches ClusterPolicy and reconciles every component across every GPU node automatically — if the driver crashes, it restarts it; if a new node joins, it installs everything on that node.

---

GPU Monitoring with DCGM

The GPU Operator deploys DCGM Exporter (Data Center GPU Manager) as a DaemonSet. It exposes per-GPU Prometheus metrics.

flowchart TD

GPU-->DCGM["DCGM <br/> (direct driver access)"]
DCGM-->Exporter["DCGM Exporter <br/> (/metrics endpoint)"]
Exporter-->Prometheus
Prometheus-->Grafana

Key metrics to monitor in production:

Metric	What it measures	Alert when
DCGM_FI_DEV_GPU_UTIL	SM utilization %	< 80% sustained (underutilization)
DCGM_FI_DEV_MEM_COPY_UTIL	Memory bandwidth utilization %	Sustained 100% (memory bound)
DCGM_FI_DEV_FB_USED	Framebuffer (HBM) memory used	> 90% of capacity
DCGM_FI_DEV_POWER_USAGE	GPU power draw (watts)	> TDP (thermal throttling)
DCGM_FI_DEV_GPU_TEMP	GPU die temperature (°C)	> 83°C (H100 throttle threshold)
DCGM_FI_DEV_ECC_DBE_VOL_TOTAL	Double-bit ECC errors	Any non-zero (hardware fault)
DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL	NVLink total bandwidth	Drop from baseline (link degraded)
DCGM_FI_DEV_PCIE_TX_THROUGHPUT	PCIe TX bytes/s	Sustained max (PCIe bottleneck)

Double-bit ECC errors (DBE) indicate a hardware fault. The GPU Operator automatically taints the node when they occur (see below).

---

Multi-Instance GPU (MIG)

An H100 SXM5 has 80 GB of HBM3 memory and 132 streaming multiprocessors (SMs).

Most inference workloads don't need all of it.

Without MIG, one Pod monopolizes the entire GPU regardless of how little of it it uses.

MIG (Multi-Instance GPU) partitions a physical GPU into isolated slices at the hardware level — each slice has its own SMs, HBM memory, and PCIe bandwidth. They cannot interfere with each other.

MIG Profiles on H100

The H100 supports the following partitions:

Profile	SMs	HBM	Max per GPU	nvidia.com resource name
1g.10gb	1/7	10 GB	7	nvidia.com/mig-1g.10gb
2g.20gb	2/7	20 GB	3	nvidia.com/mig-2g.20gb
3g.40gb	3/7	40 GB	2	nvidia.com/mig-3g.40gb
4g.40gb	4/7	40 GB	1	nvidia.com/mig-4g.40gb
7g.80gb	7/7	80 GB	1	nvidia.com/mig-7g.80gb

A common production layout for an inference node — 7 small partitions, 7 Pods per GPU:

H100 (80 GB, 132 SMs)
├── MIG 1g.10gb  → Pod: 10 GB model
├── MIG 1g.10gb  → Pod: 10 GB model
├── MIG 1g.10gb  → Pod: 10 GB model
├── MIG 1g.10gb  → Pod: 10 GB model
├── MIG 1g.10gb  → Pod: 10 GB model
├── MIG 1g.10gb  → Pod: 10 GB model
└── MIG 1g.10gb  → Pod: 10 GB model

A mixed layout for a node running both training and inference:

H100 (80 GB)
├── MIG 3g.40gb  → Training Pod (40 GB)
├── MIG 2g.20gb  → Inference Pod (20 GB)
└── MIG 2g.20gb  → Inference Pod (20 GB)

Kubernetes View of MIG

With MIG configured, the Device Plugin advertises partitions instead of whole GPUs:

Capacity:
  nvidia.com/gpu:        0           # full GPUs no longer schedulable
  nvidia.com/mig-1g.10gb: 7
  nvidia.com/mig-2g.20gb: 0
  nvidia.com/mig-3g.40gb: 1

Pods request partitions the same way they request full GPUs:

resources:
  limits:
    nvidia.com/mig-1g.10gb: "1"

The MIG Manager (deployed by GPU Operator) creates and destroys MIG partitions based on the desired configuration in a ConfigMap — no manual nvidia-smi mig commands needed.

---

GPU Sharing Strategies

Strategy	Isolation	Use case	Notes
Full GPU	Complete	Large LLM training, full-GPU inference	nvidia.com/gpu: 1
MIG	Hardware-level	Production multi-tenant inference	H100/A100 only; requires MIG Manager
Time-Slicing	None (shared context)	Development, small batch jobs	Multiple Pods share one GPU in time
Multi-Process Service (MPS)	Partial	Inference with many small requests	CUDA context shared; memory not isolated

Time-Slicing Configuration

Time-slicing is configured via a ConfigMap consumed by the Device Plugin:

apiVersion: v1
kind: ConfigMap
metadata:
  name: device-plugin-config
  namespace: gpu-operator
data:
  config.json: |
    {
      "version": "v1",
      "sharing": {
        "timeSlicing": {
          "resources": [
            {
              "name": "nvidia.com/gpu",
              "replicas": 4     # advertise each GPU as 4 schedulable units
            }
          ]
        }
      }
    }

After this, a node with 8 physical GPUs advertises nvidia.com/gpu: 32. Four Pods can share one GPU — each gets a time slice of the GPU's compute.

Important: time-slicing provides no memory isolation. A Pod that allocates all GPU memory will OOM-kill other Pods sharing the same GPU. Use MIG for production multi-tenant workloads.

---

GPU Health and Automatic Node Tainting

When a GPU develops a hardware fault, the GPU Operator automatically takes action:

flowchart TD

DCGM["DCGM detects <br/> DBE ECC error on GPU 3"]-->
MIGManager["GPU Operator <br/> health monitor"]-->
Taint["Taint node: <br/> nvidia.com/gpu-ecc-error=:NoSchedule"] 
Taint -->NewPods["New GPU Pods <br/> no longer scheduled here"]
Taint-->Alert["Prometheus alert fired <br/> → PagerDuty / Slack"]

The taint prevents new GPU Pods from landing on a degraded node. Existing Pods are not evicted — they continue running until they complete or the operator decides a reboot/replacement is needed.

DCGM also performs active health checks — it runs a short GPU compute test periodically and marks GPUs as unhealthy if the test fails, even before an ECC error appears.

Relevant taints the GPU Operator manages:

Taint	Cause
nvidia.com/gpu-driver-upgrade	Driver upgrade in progress
nvidia.com/gpu-ecc-error	Double-bit ECC hardware fault
nvidia.com/mig-config	MIG reconfiguration in progress

---

Complete Scheduling Flow

sequenceDiagram

participant User
participant API as API Server
participant Sched as Scheduler
participant Kubelet
participant Plugin as Device Plugin
participant Toolkit as nvidia-container-toolkit
participant Container

User->>API: Create Pod (nvidia.com/gpu: 1)
API->>Sched: Unscheduled Pod event
Sched->>Sched: Filter nodes (NodeResourcesFit: gpu >= 1)
Sched->>Sched: Score nodes
Sched->>API: Bind Pod → node-dgx-01
API->>Kubelet: Pod assigned (Watch event)
Kubelet->>Plugin: Allocate(GPU-3)
Plugin-->>Kubelet: /dev/nvidia3, CUDA_VISIBLE_DEVICES=3
Kubelet->>Toolkit: Create container with GPU devices
Toolkit->>Container: Inject GPU devices + CUDA libs
Container->>Container: nvidia-smi → sees GPU 3 only

---

Key Takeaways

Component	Responsibility
Device Plugin	Advertises GPU resources via ListAndWatch; allocates specific devices via Allocate
nvidia-container-toolkit	Injects /dev/nvidia* device files and CUDA libraries into the container at runtime
GPU Operator	Reconciles the full GPU software stack (driver, toolkit, plugin, monitoring) via ClusterPolicy
DCGM Exporter	Exposes per-GPU Prometheus metrics; fires alerts for ECC errors and utilization drops
MIG	Partitions one GPU into isolated hardware slices — each with its own SMs and HBM
Time-Slicing	Multiplexes one GPU across multiple Pods in time — no memory isolation
MIG Manager	Creates/destroys MIG partitions automatically based on ClusterPolicy
Health Monitor	Taints nodes on GPU fault — prevents scheduling on degraded hardware

The Kubernetes scheduler sees GPUs as opaque integer resources. Everything that makes them useful — driver compatibility, isolation, monitoring, partitioning, and health — is handled by the NVIDIA stack sitting between the scheduler's decision and the container's first CUDA call.

---

Interview Deep-Dives

🧠 1. How would you auto-scale GPU nodes for training workloads without wasting GPU hours on idle pods?

The core problem: GPU nodes are expensive — idle time costs real money. The solution layers three mechanisms.

KEDA scales the job queue, not arbitrary metrics:

apiVersion: keda.sh/v1alpha1
kind: ScaledJob
metadata:
  name: training-job-scaler
spec:
  jobTargetRef:
    template:
      spec:
        containers:
        - name: trainer
          resources:
            limits:
              nvidia.com/gpu: "8"
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus:9090
      metricName: training_queue_depth
      threshold: "1"         # one job → one node
      query: kueue_pending_workloads_total{queue="training"}
  minReplicaCount: 0         # scale to zero when queue is empty
  maxReplicaCount: 32

Cluster Autoscaler terminates idle GPU nodes:

# Node group annotation — scale down after 10 min idle
cluster-autoscaler.kubernetes.io/scale-down-delay-after-add: "10m"
cluster-autoscaler.kubernetes.io/scale-down-unneeded-time: "10m"

Kueue manages quotas so burst doesn't blow the budget:

apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
  name: a100-80gb
spec:
  nodeLabels:
    nvidia.com/gpu.product: "A100-SXM4-80GB"
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: training-queue
spec:
  resourceGroups:
  - coveredResources: ["nvidia.com/gpu"]
    flavors:
    - name: a100-80gb
      resources:
      - name: nvidia.com/gpu
        nominalQuota: 64
        borrowingLimit: 32   # can borrow from other teams

The full loop: Kueue holds jobs until GPUs are available → KEDA sees pending queue depth → Cluster Autoscaler provisions GPU nodes → jobs run → nodes idle for 10 min → Cluster Autoscaler terminates them → cost stops.

---

🧠 2. A multi-cluster, multi-region AI training job fails halfway because one cluster runs out of GPU memory. How do you rebalance workloads live?

This is a multi-cluster GPU OOM mid-training scenario. Three layers to address it.

Detection — DCGM fires before OOM:

# PrometheusRule fires at 90% VRAM — before OOM kills
- alert: GPUMemoryPressureCritical
  expr: |
    DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_TOTAL > 0.90
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "GPU memory at {{ $value | humanizePercentage }} on {{ $labels.instance }}"

Checkpoint — save state before migrating:

PyTorch distributed training must checkpoint before any node goes away:

# Elastic training with torch.distributed.elastic
# Checkpoints every N steps to shared storage (S3 / NFS)
if step % checkpoint_interval == 0:
    dist.barrier()                          # all ranks sync
    if dist.get_rank() == 0:
        torch.save(model.state_dict(), f"s3://checkpoints/step-{step}.pt")

Live rebalancing with Volcano gang scheduling:

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: distributed-training
spec:
  minAvailable: 16        # minimum ranks to continue
  plugins:
    pytorch: ["--master_port=23456"]
  tasks:
  - replicas: 8
    name: worker
    template:
      spec:
        containers:
        - name: trainer
          env:
          - name: RDZV_BACKEND
            value: etcd           # rendezvous survives node loss
          resources:
            limits:
              nvidia.com/gpu: "8"

Cross-cluster rescheduling with Admiralty / Liqo:

Admiralty projects GPU capacity from a remote cluster as virtual nodes — the scheduler sees them as local. When cluster-A OOMs, pending jobs automatically land on cluster-B's virtual nodes.

apiVersion: multicluster.admiralty.io/v1alpha1
kind: Target
metadata:
  name: cluster-b-us-west
spec:
  kubeconfigSecret:
    name: cluster-b-kubeconfig

The rebalance flow: DCGM alert fires at 90% VRAM → checkpoint saves to S3 → Volcano reschedules surviving ranks → Admiralty routes new pods to cluster-B → training resumes from last checkpoint with zero data loss.

---

🧠 3. How do you configure Kubernetes taints and tolerations for GPU workloads?

Taints prevent non-GPU workloads from landing on expensive GPU nodes. Tolerations let GPU pods opt in.

Taint GPU nodes at provision time:

# Apply taint when node joins
kubectl taint nodes gpu-node-01 nvidia.com/gpu=present:NoSchedule

# Or via node group label in Cluster Autoscaler config
taints:
  - key: nvidia.com/gpu
    value: present
    effect: NoSchedule

GPU Operator adds taints automatically when it detects a GPU node — no manual step needed:

# ClusterPolicy — GPU Operator manages the taint
apiVersion: nvidia.com/v1
kind: ClusterPolicy
spec:
  daemonsets:
    tolerations:
    - key: nvidia.com/gpu
      operator: Exists
      effect: NoSchedule

GPU pod toleration:

spec:
  tolerations:
  - key: nvidia.com/gpu
    operator: Exists
    effect: NoSchedule
  nodeSelector:
    nvidia.com/gpu.product: "A100-SXM4-80GB"   # target specific GPU SKU
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels:
            app: training-job
        topologyKey: kubernetes.io/hostname     # one job per node

Multi-GPU taint strategy by workload type:

# Training nodes — exclusive, high-memory GPUs
kubectl taint nodes -l workload=training  dedicated=training:NoSchedule

# Inference nodes — shared, time-sliced GPUs
kubectl taint nodes -l workload=inference dedicated=inference:NoSchedule

# Dev nodes — MIG-sliced, shared
kubectl taint nodes -l workload=dev       dedicated=dev:NoSchedule

Each team's namespace gets a LimitRange that only tolerates their node tier — preventing training jobs from accidentally landing on inference nodes and vice versa.

---

🧠 4. How would you handle CUDA driver upgrades in Kubernetes without disrupting thousands of running AI pods?

CUDA driver upgrades are one of the hardest operational problems in GPU clusters — a driver mismatch kills every container on the node.

Rule 1 — never upgrade drivers on a node with running pods.

# Cordon + drain before any driver touch
kubectl cordon gpu-node-01
kubectl drain gpu-node-01 \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --grace-period=300          # 5-min graceful shutdown

For training jobs, pair drain with PodDisruptionBudget:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: training-pdb
spec:
  maxUnavailable: 1           # drain one node at a time
  selector:
    matchLabels:
      app: distributed-training

Rule 2 — rolling upgrade via node groups.

In cloud environments (EKS, GKE, AKS) use node group rolling replace:

# EKS managed node group — replace 1 node at a time
aws eks update-nodegroup-config \
  --cluster-name prod \
  --nodegroup-name gpu-a100 \
  --update-config maxUnavailable=1

Rule 3 — driver upgrade without node replacement using GPU Operator.

GPU Operator can upgrade drivers in-place via the Driver DaemonSet:

apiVersion: nvidia.com/v1
kind: ClusterPolicy
spec:
  driver:
    version: "550.90.07"       # bump version here
    upgradePolicy:
      autoUpgrade: true
      maxParallelUpgrades: 1   # one node at a time
      drain:
        enable: true
        force: false
        podSelector: ""        # drain all pods
        timeoutSeconds: 300
        deleteEmptyDir: true

GPU Operator drains the node, upgrades the driver, validates with nvidia-smi, then uncordons — without any manual SSH.

Rule 4 — CUDA forward compatibility for zero-downtime.

NVIDIA's forward compatibility layer lets containers built against newer CUDA toolkit run on older drivers — breaking the tight driver/container coupling:

FROM nvcr.io/nvidia/cuda:12.4.0-base-ubuntu22.04
# This image works on drivers ≥ 520.x via forward compat

The safe upgrade sequence: announce maintenance window → drain nodes one at a time behind PDB → GPU Operator upgrades driver → validate with nvidia-smi and a smoke test pod → uncordon → move to next node.

---

🧠 5. How would you pre-warm GPU nodes for massive AI inference traffic with zero cold-start penalty?

At ChatGPT scale, cold-start means: node spin-up (3–5 min) + driver init + model load (can be minutes for 70B+ models). Pre-warming eliminates all of this.

Layer 1 — Keep minimum replicas hot:

# HPA never scales below 2 — always at least 2 warm replicas
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
  minReplicas: 2
  maxReplicas: 100
  metrics:
  - type: External
    external:
      metric:
        name: inference_queue_depth
      target:
        type: AverageValue
        averageValue: "10"

Layer 2 — Pre-provision GPU nodes before traffic arrives:

KEDA with a cron trigger spins up nodes ahead of peak traffic:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
spec:
  triggers:
  - type: cron
    metadata:
      timezone: Europe/Berlin
      start: "0 8 * * 1-5"      # scale up at 8am weekdays
      end:   "0 20 * * 1-5"     # scale down at 8pm
      desiredReplicas: "20"
  - type: prometheus              # also react to live load
    metadata:
      query: inference_rps
      threshold: "100"

Layer 3 — Pre-load model weights using init containers:

spec:
  initContainers:
  - name: model-loader
    image: aws-cli
    command:
    - aws s3 cp s3://models/llama-70b/ /model-cache/ --recursive
    volumeMounts:
    - name: model-cache
      mountPath: /model-cache
  containers:
  - name: inference-server
    image: nvcr.io/nvidia/nim/llama3:70b
    env:
    - name: NIM_MODEL_PATH
      value: /model-cache          # weights already on disk
    volumeMounts:
    - name: model-cache
      mountPath: /model-cache
  volumes:
  - name: model-cache
    hostPath:
      path: /mnt/model-cache       # persists across pod restarts
      type: DirectoryOrCreate

Layer 4 — Node image pre-pull with DaemonSet:

# Force image pull on every GPU node before pods need it
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: image-prepuller
spec:
  selector:
    matchLabels:
      app: image-prepuller
  template:
    spec:
      nodeSelector:
        nvidia.com/gpu: "true"
      initContainers:
      - name: pull-nim
        image: nvcr.io/nvidia/nim/llama3:70b
        command: ["true"]          # just pulls the image, exits
      containers:
      - name: pause
        image: gcr.io/google_containers/pause:3.1

Layer 5 — Readiness gate — only serve when model is loaded:

readinessProbe:
  httpGet:
    path: /v1/models              # NIM readiness endpoint
    port: 8000
  initialDelaySeconds: 60        # wait for model load
  periodSeconds: 10
  failureThreshold: 30           # 5 min total wait

The result: GPU node already running → container image already pulled → model weights already on disk → inference server warm → traffic hits a ready pod within seconds, not minutes.

---

🧠 6. How would you monitor GPU utilization in real-time in a Kubernetes cluster?

The full observability stack: DCGM → Prometheus → Grafana → Alertmanager.

Step 1 — Deploy DCGM Exporter via GPU Operator:

GPU Operator installs the DCGM Exporter DaemonSet automatically. Metrics are scraped at :9400/metrics.

Step 2 — Key metrics to track:

# GPU utilization % per pod
DCGM_FI_DEV_GPU_UTIL{pod=~"training-.*"}

# VRAM used vs total
DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_TOTAL * 100

# GPU temperature (thermal throttling risk)
DCGM_FI_DEV_GPU_TEMP > 80

# NVLink bandwidth (bottleneck in multi-GPU training)
DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL

# XID errors (hardware faults)
DCGM_FI_DEV_XID_ERRORS > 0

# Tensor Core active cycles (are you using mixed precision?)
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE

Step 3 — Prometheus scrape config:

# ServiceMonitor for GPU Operator's DCGM Exporter
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: dcgm-exporter
spec:
  selector:
    matchLabels:
      app: dcgm-exporter
  endpoints:
  - port: metrics
    interval: 15s
    path: /metrics

Step 4 — Grafana dashboard:

Import dashboard ID 12239 (NVIDIA DCGM Exporter) — pre-built panels for:

• Per-GPU utilization heatmap across all nodes
• VRAM pressure per workload
• Temperature per GPU
• XID error timeline
• NVLink throughput

Step 5 — Alerts that matter:

groups:
- name: gpu-alerts
  rules:
  - alert: GPUIdleReservation
    expr: |
      DCGM_FI_DEV_GPU_UTIL < 10
      and on(pod) kube_pod_status_phase{phase="Running"} == 1
    for: 30m
    annotations:
      summary: "Pod {{ $labels.pod }} has reserved GPU but utilization is {{ $value }}%"

  - alert: GPUMemoryCritical
    expr: DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_TOTAL > 0.95
    for: 5m
    annotations:
      summary: "GPU memory at {{ $value | humanizePercentage }} — OOM imminent"

  - alert: GPUHardwareFault
    expr: DCGM_FI_DEV_XID_ERRORS > 0
    for: 0m
    labels:
      severity: critical
    annotations:
      summary: "XID error {{ $value }} on {{ $labels.instance }} — check hardware"

  - alert: GPUTemperatureHigh
    expr: DCGM_FI_DEV_GPU_TEMP > 85
    for: 5m
    annotations:
      summary: "GPU temperature {{ $value }}°C on {{ $labels.instance }}"

Step 6 — Cost visibility with Kubecost:

helm install kubecost kubecost/cost-analyzer \
  --set prometheus.enabled=false \
  --set global.prometheus.enabled=false \
  --set global.prometheus.fqdn=http://prometheus:9090

Kubecost reads DCGM utilization and correlates it with node costs — surfacing idle GPU spend per namespace, team, and job.

---

🧠 7. What happens when a pod requests 8 GPUs but no node has more than 6?

The pod goes immediately to Pending — it is never placed. GPUs are never split across nodes for a single container. The scheduler's filter phase finds zero qualifying nodes and the pod waits indefinitely.

kubectl describe pod gpu-job
# Events:
#   Warning  FailedScheduling  0/12 nodes are available:
#            4 nodes have 6 nvidia.com/gpu allocatable,
#            8 nodes have insufficient nvidia.com/gpu.

What happens in each scenario:

Cluster state	Outcome
No node has ≥ 8 GPUs (hardware limit)	Pending forever
Node has 8 GPUs but 2 are in use	Pending until 2 pods release their GPUs
Cluster Autoscaler + 8-GPU node group exists	New node provisioned in 3–5 min, pod scheduled
Kueue quota < 8 GPUs	Held in queue until quota frees up
Node has 8 GPUs but taint not tolerated	Pending — node has taint

The request is all-or-nothing. Even if the cluster has 48 free GPUs spread across 8 six-GPU nodes, a single pod requesting 8 gets nothing — no single node satisfies it.

If you genuinely need 8 GPUs across two 4-GPU nodes, you need a distributed Job with one pod per node, not a single pod:

apiVersion: batch/v1
kind: Job
spec:
  completions: 2
  parallelism: 2
  template:
    spec:
      containers:
      - name: trainer
        resources:
          limits:
            nvidia.com/gpu: "4"   # 4 per pod × 2 pods = 8 total GPUs
        env:
        - name: WORLD_SIZE
          value: "2"

Use Volcano or Kueue for gang scheduling — so both pods start together or neither starts:

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
spec:
  minAvailable: 2     # all-or-nothing placement

Cluster Autoscaler behaviour: CA detects the unschedulable pod, simulates whether a new node would fix it, and provisions one if the node group supports 8+ GPUs. If the largest node group only has 6-GPU nodes, CA gives up — pod stays Pending.

How to diagnose:

# Why is it pending?
kubectl describe pod <pod> | grep -A 10 Events

# What GPU capacity exists per node?
kubectl get nodes -o custom-columns=\
"NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu"

# How many GPUs already in use?
kubectl describe nodes | grep -A 5 "Allocated resources" | grep gpu

---

🧠 8. How would you check if a GPU pod in Kubernetes is actually using the GPU assigned to it?

There are three layers to verify: device assignment, driver visibility, and actual compute utilisation.

Layer 1 — Verify the device is assigned:

# Check which GPU device files are mounted into the container
kubectl exec -it <pod> -- ls /dev/nvidia*
# Expected: /dev/nvidia0, /dev/nvidiactl, /dev/nvidia-uvm

# Check CUDA_VISIBLE_DEVICES set by the device plugin
kubectl exec -it <pod> -- env | grep CUDA_VISIBLE_DEVICES
# CUDA_VISIBLE_DEVICES=2   ← pod sees only GPU index 2

Layer 2 — Verify the driver sees the GPU:

kubectl exec -it <pod> -- nvidia-smi
# Should show the assigned GPU with memory and utilisation

If nvidia-smi errors: the nvidia-container-toolkit is misconfigured or the pod is missing the GPU toleration.

Layer 3 — Verify actual compute utilisation:

# Real-time utilisation from inside the pod
kubectl exec -it <pod> -- nvidia-smi dmon -s u -d 1
# sm   mem   enc   dec   jpg   ofa
# 87    92     0     0     0     0   ← SM utilisation 87%, VRAM 92%

# From the node — see which PID owns the GPU
kubectl exec -it <pod> -- nvidia-smi --query-compute-apps=pid,used_memory \
  --format=csv,noheader

Layer 4 — Prometheus confirms at cluster level:

# Utilisation for a specific pod
DCGM_FI_DEV_GPU_UTIL{pod="<pod-name>"}

# Detect ghost allocations — GPU reserved but idle > 10 min
DCGM_FI_DEV_GPU_UTIL{pod=~".+"} < 5

Common failure modes:

Symptom	Root cause
nvidia-smi not found	nvidia-container-toolkit not installed
CUDA_VISIBLE_DEVICES=""	Device plugin failed to allocate
nvidia-smi shows GPU but util=0	Process not actually running kernels
Wrong GPU index visible	Device plugin bug or node-level misconfiguration

---

🧠 9. What are NCCL logs and why are they important in distributed training?

NCCL (NVIDIA Collective Communications Library) handles all GPU-to-GPU communication in distributed training — AllReduce, AllGather, Broadcast. When training is slow or hangs, NCCL logs are the first place to look.

Enable NCCL logging:

# In your pod spec or training script
export NCCL_DEBUG=INFO          # topology + algorithm selection
export NCCL_DEBUG=WARN          # errors and warnings only
export NCCL_DEBUG=TRACE         # every operation (very verbose)
export NCCL_DEBUG_FILE=/tmp/nccl-rank-%h-%p.log  # per-rank log file

What NCCL INFO logs tell you:

NCCL INFO Bootstrap: Using eth0:10.0.0.5<0>
NCCL INFO Channel 00/02: 0[0] -> 1[1] -> 2[0] -> 3[1] -> 0[0]
NCCL INFO Ring 00: GPU ranks: 0 1 2 3
NCCL INFO Using network IB/ROCE    ← InfiniBand selected (good)
# vs
NCCL INFO Using network Socket     ← falling back to TCP (bad — 10x slower)

Key things to look for:

Log entry	What it means
Using network IB/ROCE	Fast path — InfiniBand or RoCE in use
Using network Socket	Slow path — TCP fallback, check RDMA config
Channel 00/02	2 rings — bandwidth doubles with more rings
WARN Timeout	Rank not responding — check pod liveness
WARN Mismatch in collective	Different ranks entered different collectives — code bug
trees vs rings	NCCL chose tree for large messages (lower latency)

NCCL environment variables that affect training performance:

# Force InfiniBand (never fall back to TCP)
export NCCL_IB_DISABLE=0
export NCCL_SOCKET_IFNAME=^lo,docker  # exclude loopback

# Tune for DGX H100 (NVLink + InfiniBand)
export NCCL_P2P_LEVEL=NVL      # use NVLink for intra-node
export NCCL_IB_HCA=mlx5        # use Mellanox HCA for inter-node
export NCCL_NET_GDR_LEVEL=2    # GPU Direct RDMA level

# Increase ring buffer for large AllReduce
export NCCL_BUFFSIZE=33554432  # 32MB per channel

NCCL hang diagnosis:

# Hang = one rank stuck, others waiting
# Step 1: find which rank is slow
grep "WARN" /tmp/nccl-rank-*.log

# Step 2: check if it's a network issue
kubectl exec -it <pod> -- ib_send_bw -d mlx5_0   # InfiniBand bandwidth test

# Step 3: check if NVLink is degraded
nvidia-smi nvlink --status -i 0

# Step 4: nuclear option — dump all rank stack traces
kubectl exec -it <pod> -- kill -USR1 <training-pid>

Why NCCL logs matter in interviews: they're the canonical way to diagnose the most common distributed training problems — network fallback to TCP (10x throughput loss), ring topology misconfiguration, and rank hangs that silently stall 512-GPU jobs.

---

🧠 10. Persistent storage for AI datasets shows 200ms+ latency. How do you pinpoint whether it's the storage backend, the network, or the GPU node?

Isolate each layer with targeted benchmarks — never assume.

Step 1 — Baseline from outside Kubernetes (eliminate k8s overhead):

# SSH directly to the storage node or NFS server
fio --name=read-test \
    --filename=/mnt/dataset/test.bin \
    --rw=randread \
    --bs=4k \
    --numjobs=8 \
    --iodepth=32 \
    --runtime=30 \
    --time_based \
    --output-format=json
# If latency is high here → storage backend problem

Step 2 — Test network path between GPU node and storage:

# Run inside the GPU pod
kubectl exec -it <pod> -- \
  fio --name=network-test \
      --filename=/data/test.bin \
      --rw=randread --bs=4k --numjobs=4 \
      --iodepth=16 --runtime=30 --time_based

# Compare with Step 1 result
# If Step 2 latency >> Step 1 latency → network is the bottleneck

Step 3 — Isolate network with raw throughput test:

# iperf3 between GPU node and storage server
kubectl exec -it <pod> -- iperf3 -c <storage-server-ip> -t 30 -P 8

# Check for packet loss and retransmits
kubectl exec -it <pod> -- ss -s | grep retrans

# Check NIC errors on the GPU node
kubectl debug node/<node> -it --image=nicolaka/netshoot -- \
  ethtool -S eth0 | grep -i error

Step 4 — Check if it's I/O wait on the GPU node itself:

# High iowait means the node CPU is waiting for storage
kubectl debug node/<node> -it --image=nicolaka/netshoot -- \
  iostat -x 1 10

# Check if the NVMe local disk is being hit instead of the NFS mount
kubectl debug node/<node> -it --image=nicolaka/netshoot -- \
  iotop -o

Step 5 — Prometheus metrics to correlate with the 200ms spike:

# NFS read latency
node_nfs_requests_total

# Network receive errors on storage interface
rate(node_network_receive_errs_total[5m])

# Disk await time
rate(node_disk_read_time_seconds_total[5m])
  / rate(node_disk_reads_completed_total[5m])

Decision matrix:

Observation	Root cause	Fix
High latency in Step 1	Storage backend (disk, NFS server)	Scale IOPS, check disk health
Low latency in Step 1, high in Step 2	Network path	Check MTU, NIC errors, switch congestion
High iowait on GPU node	Local disk I/O contention	Move dataset cache to tmpfs or local NVMe
Latency spikes correlated with training step	GPU pod DataLoader is synchronous	Use async prefetching, increase num_workers
Latency only on one node	Bad NIC or faulty switch port	Cordon node, replace hardware

Quick win for AI dataset latency: cache the dataset on local NVMe using a DaemonSet prepuller before training starts — completely eliminates network storage latency during training.

---

🧠 11. A Kubernetes GPU pod requests 16GB VRAM but only gets 12GB due to fragmentation. How do you detect and fix this in real-time?

First: clarify the problem. Kubernetes GPU allocation is integer-based — it does not track VRAM in bytes. When you request nvidia.com/gpu: 1, you get the whole GPU. If you're only seeing 12GB of a 16GB GPU, the VRAM is being consumed by something else on that GPU.

Detect what's consuming the missing 4GB:

# See all processes using GPU memory on the node
kubectl debug node/<node> -it --image=ubuntu -- nvidia-smi

# Output:
# GPU   PID   Type   Process name        GPU Memory Usage
#   0  1234    C    python training.py      12288MiB
#   0  5678    C    zombie-process.py        4096MiB  ← this is your leak

# DCGM breakdown per process
kubectl exec -it <dcgm-exporter-pod> -- dcgmi dmon -e 252,1004 -d 1
# Field 252 = FB_USED (total VRAM used)
# Field 1004 = VRAM used by compute processes

Common causes of "missing" VRAM:

Cause	How to detect	Fix
Previous pod didn't release GPU memory (zombie process)	nvidia-smi shows PID from dead container	kubectl delete pod + verify PID gone
CUDA context not destroyed on pod exit	Stale PIDs in nvidia-smi	Restart kubelet on node: systemctl restart kubelet
MIG mode partially configured	nvidia-smi -L shows unexpected MIG instances	Reset MIG: nvidia-smi mig -dci && nvidia-smi mig -dgi
Driver-level fragmentation	Fragmented after many alloc/free cycles	Node reboot (last resort)

Real-time fix without rebooting:

# Step 1: identify the zombie PID
nvidia-smi --query-compute-apps=pid,used_memory --format=csv

# Step 2: kill it
kubectl debug node/<node> -it --image=ubuntu -- kill -9 <zombie-pid>

# Step 3: verify VRAM released
kubectl debug node/<node> -it --image=ubuntu -- nvidia-smi --query-gpu=memory.free --format=csv

# Step 4: if PID won't die, reset the GPU (brief service interruption)
nvidia-smi --gpu-reset -i 0

Prevention — add VRAM check to readiness probe:

readinessProbe:
  exec:
    command:
    - sh
    - -c
    - |
      FREE=$(nvidia-smi --query-gpu=memory.free --format=csv,noheader,nounits | head -1)
      [ "$FREE" -gt 14000 ]   # fail if less than 14GB free
  initialDelaySeconds: 10
  periodSeconds: 30

Alert on fragmentation before it hits pods:

# Fire when free VRAM drops below expected for an idle GPU
(DCGM_FI_DEV_FB_TOTAL - DCGM_FI_DEV_FB_USED) < 14000
and DCGM_FI_DEV_GPU_UTIL < 5

---

🧠 12. Your AI pipeline cost doubles in 24 hours with no infra change. Profiling shows a silent GPU resource leak. How do you hunt it down?

Cost doubling with no code change points to one of three things: more GPUs allocated, GPUs running longer than expected, or a ghost allocation that never releases.

Step 1 — Confirm the cost source with Kubecost:

# Which namespace/deployment is consuming the extra GPU hours?
kubectl cost namespace --window 24h --show-cpu --show-memory --show-gpu

# GPU hours consumed per namespace over 24h
sum_over_time(
  count(DCGM_FI_DEV_GPU_UTIL{namespace!=""} > 0)[24h:1m]
) by (namespace) / 60

Step 2 — Find pods holding GPUs but doing no work:

# Pods with GPU allocated but near-zero utilisation for > 30 min
DCGM_FI_DEV_GPU_UTIL < 5
and on(pod) kube_pod_status_phase{phase="Running"} == 1

# List all pods currently holding GPUs
kubectl get pods -A -o json | jq -r '
  .items[] |
  select(.spec.containers[].resources.limits["nvidia.com/gpu"] != null) |
  [.metadata.namespace, .metadata.name,
   .spec.containers[].resources.limits["nvidia.com/gpu"],
   .status.phase] | @tsv'

Step 3 — Hunt the leak pattern:

# Check if Jobs completed but pods are stuck in Completed/Error state still holding GPU
kubectl get pods -A --field-selector=status.phase=Succeeded | grep gpu

# Check for pods with no owner (orphaned — never cleaned up)
kubectl get pods -A -o json | jq -r '
  .items[] |
  select(.metadata.ownerReferences == null) |
  [.metadata.namespace, .metadata.name, .status.phase] | @tsv'

# Check TTL on Jobs — if not set, completed pods linger
kubectl get jobs -A -o json | jq -r '
  .items[] |
  select(.spec.ttlSecondsAfterFinished == null) |
  .metadata.name'

Step 4 — Check for runaway autoscaling:

# Did HPA or KEDA scale up and never scale back?
kubectl get hpa -A
kubectl describe hpa <name> | grep -E "Min|Max|Current|Desired"

# Check Cluster Autoscaler added nodes that never scaled back
kubectl get nodes --sort-by=.metadata.creationTimestamp | tail -20

Step 5 — CUDA-level leak inside the container:

# Track VRAM usage growth over time for a specific pod
watch -n 5 'kubectl exec -it <pod> -- \
  nvidia-smi --query-gpu=memory.used --format=csv,noheader'

# If memory grows monotonically with no release → CUDA memory leak in code
# Profile with:
kubectl exec -it <pod> -- \
  compute-sanitizer --tool memcheck python train.py

Fix checklist once the leak is found:

# 1. Set TTL on all Jobs — auto-delete after completion
spec:
  ttlSecondsAfterFinished: 300   # delete 5 min after job ends

# 2. Set activeDeadlineSeconds — kill runaway jobs
spec:
  activeDeadlineSeconds: 86400   # max 24h job runtime

# 3. ResourceQuota cap on GPU — limits blast radius
spec:
  hard:
    requests.nvidia.com/gpu: "32"

# 4. KEDA scale-to-zero — idle replicas cost nothing
spec:
  minReplicaCount: 0

Alert to catch it next time in < 5 minutes:

# GPU cost rate doubles vs 1-hour rolling average
sum(DCGM_FI_DEV_GPU_UTIL > 0) by (cluster)
  > 2 * avg_over_time(sum(DCGM_FI_DEV_GPU_UTIL > 0) by (cluster)[1h])

---

Related Posts

• AI Infra Computing: GPU, DPU, Virtualization, DGX Systems — the hardware underneath: H100 SM architecture, MIG hardware partitioning, DGX system specs, and vGPU vs MIG comparison
• AI Programming Model: CUDA — how CUDA kernels, warps, and streaming multiprocessors work — what nvidia-container-toolkit injects access to
• Dynamic Resource Allocation: The Future of GPU Scheduling — the replacement for Device Plugins: ResourceClaim, CEL selectors, and topology-aware GPU allocation
• Kubernetes Topology Manager: NUMA-Aware GPU Scheduling — how the kubelet co-locates the GPU, CPU, and NIC on the same NUMA node after the scheduler selects the node
• Cloud Native Observability — how DCGM Exporter feeds GPU metrics into Prometheus and Grafana for the dashboards referenced in this post