Cloud Native Observability: Prometheus, Grafana, OpenTelemetry, and Tracing

A Kubernetes cluster running hundreds of Pods across dozens of nodes can fail in ways that are invisible unless you're watching for them.

Observability is the practice of making a system's internal state inferable from its external outputs.

Monitoring vs Observability

Monitoring	Observability
Detect known issues	Investigate unknown issues
Dashboards	End-to-end visibility
Alerts	Root cause analysis
Metrics focused	Metrics + Logs + Traces
Reactive	Proactive

The Four Golden Signals

Google SRE recommends monitoring:

Signal	Meaning
Latency	Request duration
Traffic	Request volume
Errors	Failed requests
Saturation	Resource utilization

RED Method

Used for microservices.

Metric	Description
Rate	Requests per second
Errors	Failed requests
Duration	Request latency

USE Method

Used for infrastructure.

Metric	Description
Utilization	Resource usage
Saturation	Resource contention
Errors	Hardware/software errors

---

Signal Types

In distributed systems there are three primary signal types — called the three pillars:

flowchart LR

Metrics["📊 Metrics <br/> What is happening? <br/> (numbers over time)"]

Logs["📄 Logs <br/> What happened? <br/> (events with context)"]

Traces["🔗 Traces <br/> Where did it happen? <br/> (request flow across services)"]

Each pillar answers a different question.

• Metrics alert you that something is wrong.
• Logs tell you what the error was.
• Traces show you which service in a chain caused it.

Let's explore all 3 in details

1. METRICS 📊

Metrics are numeric measurements collected over time.

Metric Types

Type	What it represents	Example
Counter	Monotonically increasing value (never decreases)	Total HTTP requests, total errors
Gauge	Value that can go up or down	Current memory usage, queue depth
Histogram	Distribution of values in configurable buckets	Request latency p50/p90/p99
Summary	Pre-computed quantiles (client-side)	Similar to Histogram but less flexible

# Counter
    http_requests_total{method="GET", status="200"} 12483
    
    # Gauge
    kube_pod_status_ready{pod="nim-llm-abc", namespace="inference"} 1
    
    # Histogram (multiple lines — one per bucket)
    apiserver_request_duration_seconds_bucket{verb="GET", le="0.1"}  8432
    apiserver_request_duration_seconds_bucket{verb="GET", le="0.5"}  9211
    apiserver_request_duration_seconds_bucket{verb="GET", le="1.0"}  9398
    apiserver_request_duration_seconds_count{verb="GET"}             9401
    apiserver_request_duration_seconds_sum{verb="GET"}               523.4

Exporters

Applications expose metrics using exporters.

Examples

• Node Exporter
• kube-state-metrics
• cAdvisor
• PostgreSQL Exporter
• Redis Exporter
• NVIDIA DCGM Exporter

Prometheus

Prometheus is the de-facto standard for metrics in Kubernetes.

Responsibilities

• Scrape metrics
• Store time-series data
• Execute PromQL queries
• Trigger alerts

sequenceDiagram
    participant Prometheus
    participant Pod

    Prometheus->>Pod: GET /metrics

    Pod-->>Prometheus: Prometheus Metrics

It is a pull-based time-series database: Prometheus scrapes HTTP endpoints that expose metrics, stores them, and makes them queryable via PromQL.

flowchart TD

    App["Application <br/> /metrics endpoint <br/> (Prometheus format)"]
    App-->|"HTTP scrape <br/> every 15s"| Prometheus
    Prometheus-->|"PromQL query"| Grafana

    Prometheus-->|"alert rule"| Alertmanager
    Alertmanager-->|"notification"| PagerDuty["PagerDuty <br/> Slack <br/> Email"]

PromQL — Prometheus Query Language

PromQL is how you ask questions of your metrics data:

# Rate of HTTP requests per second over last 5 minutes
rate(http_requests_total[5m])

# p99 API Server latency
histogram_quantile(0.99,
  rate(apiserver_request_duration_seconds_bucket[5m])
)

# GPU utilization averaged per node
avg by (node) (DCGM_FI_DEV_GPU_UTIL)

# Pods not in Running state
kube_pod_status_phase{phase!="Running"} == 1

# Free GPU memory per node
DCGM_FI_DEV_FB_FREE / DCGM_FI_DEV_FB_TOTAL * 100

kube-prometheus-stack

The standard way to deploy Prometheus on Kubernetes is the kube-prometheus-stack Helm chart — it bundles Prometheus, Alertmanager, Grafana, and a set of pre-built dashboards and alert rules:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm upgrade --install kube-prometheus-stack \
  prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace \
  -f monitoring-values.yaml

It deploys the Prometheus Operator, which introduces ServiceMonitor and PodMonitor CRDs — instead of editing Prometheus config files, you declare what to scrape with Kubernetes objects:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: dcgm-exporter
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: dcgm-exporter        # matches the DCGM Exporter Service
  namespaceSelector:
    matchNames: [gpu-operator]
  endpoints:
  - port: metrics
    interval: 15s
    path: /metrics

Prometheus automatically discovers and scrapes any Service that this ServiceMonitor matches — no manual config reload needed.

Alert Manager

Alertmanager handles routing and deduplication of alerts fired by Prometheus alert rules:

# PrometheusRule CRD — defines alert conditions
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: gpu-alerts
  namespace: monitoring
spec:
  groups:
  - name: gpu
    rules:
    - alert: GPUHighTemperature
      expr: DCGM_FI_DEV_GPU_TEMP > 83
      for: 2m
      labels:
        severity: warning
      annotations:
        summary: "GPU temperature above throttle threshold on {{ $labels.pod }}"

    - alert: GPUECCDoublebitError
      expr: increase(DCGM_FI_DEV_ECC_DBE_VOL_TOTAL[10m]) > 0
      labels:
        severity: critical
      annotations:
        summary: "GPU hardware ECC double-bit error — node may need replacement"

Grafana

Grafana is the visualization layer.

It connects to multiple data sources (Prometheus, Loki, Tempo) and renders dashboards.

flowchart TD

Prometheus-->|"PromQL"| Grafana
Loki["Loki <br/> (logs)"]-->|"LogQL"| Grafana
Tempo["Tempo <br/> (traces)"]-->|"TraceQL"| Grafana
Grafana-->|"render"| Dashboard["Dashboards <br/> Panels <br/> Alerts"]

Key Grafana concepts:

Concept	What it is
Data source	Connection to a backend (Prometheus, Loki, Tempo, Elasticsearch)
Dashboard	Collection of panels arranged on a grid
Panel	A single visualization (graph, gauge, table, heatmap) with a query
Variable	Dropdown filter that parameterizes queries (e.g., $namespace, $node)
Alert	Grafana can fire alerts directly from dashboard panels

GPU Dashboard on DGX

A typical GPU monitoring dashboard has panels for:

• GPU utilization % (per GPU, heatmap across all nodes)
• HBM memory used / free
• GPU temperature with 83°C threshold line
• NVLink bandwidth (training throughput indicator)
• DCGM ECC error count (hardware health)
• Power draw vs TDP

The NVIDIA DCGM Exporter ships with pre-built Grafana dashboard JSON that imports directly into Grafana.

---

2. LOGS

Logs record discrete events.

Why Structured Logging

Unstructured logs are for humans. Structured logs are for machines.

# Unstructured — hard to filter, parse, or aggregate
[2026-07-07 14:32:01] ERROR: connection to database failed after 3 retries

# Structured JSON — queryable, indexable, aggregatable
{"timestamp":"2026-07-07T14:32:01Z","level":"error","msg":"connection failed",
 "retries":3,"service":"api","pod":"api-7d9f8b-xkp2m","namespace":"prod"}

Structured logs let you filter across thousands of Pods with a single query.

Fluent Bit

Fluent Bit runs as a DaemonSet on every node.

Collects

• Container logs
• Kubernetes metadata
• System logs

Forwards them to

• Loki
• Elasticsearch
• Splunk

It tails container log files from /var/log/containers/, parses them, and ships them to a log backend.

flowchart TD

    Containers["Container logs <br/> /var/log/containers/*.log"]
    Containers-->FluentBit["Fluent Bit <br/> (DaemonSet — one per node)"]

    FluentBit-->|"forward"| Loki["Loki <br/> (log aggregator)"]
    Loki-->|"LogQL query"| Grafana

helm upgrade --install fluent-bit fluent/fluent-bit \
  --namespace logging --create-namespace \
  --set config.outputs="[OUTPUT] <br/>     Name loki <br/>     Host loki.monitoring.svc.cluster.local"

Loki

Loki is a log aggregation system designed by Grafana Labs.

It indexes only labels (like Prometheus), not the full log content — this makes it dramatically cheaper than Elasticsearch for Kubernetes log volumes.

# LogQL — query logs like PromQL queries metrics

# All error logs from the inference namespace
{namespace="inference"} |= "error"

# NIM Pod logs filtered to HTTP 5xx
{app="llama3-70b-nim"} | json | status >= 500

# Rate of error logs per Pod over 5 minutes
rate({namespace="training"} |= "ERROR" [5m])

# Training loss from structured log field
{app="pytorch-worker"} | json | __error__="" | unwrap loss [1m]

Loki stores log streams as compressed chunks on object storage (S3, GCS) — far cheaper than Elasticsearch for the same retention period.

---

3. TRACE 🔗

A trace follows a request through multiple services.

A single user request in a microservices system touches many services. If the request is slow or fails, which service is responsible?

Distributed tracing answers this by following a request across service boundaries.

flowchart LR

    Request["User request <br/> trace_id=abc123"]

    Request-->GW["API Gateway <br/> span: 5ms"]
    GW-->Auth["Auth Service <br/> span: 12ms"]
    GW-->Model["Model Service <br/> span: 340ms"]

    Model-->Cache["Cache <br/> span: 2ms"]
    Model-->GPU["GPU Inference <br/> span: 320ms"]

The full trace shows the request took 357 ms total, with 320 ms spent in GPU inference — the bottleneck is identified immediately.

Key Concepts

Concept	What it is
Trace	The complete journey of one request across all services
Span	A single operation within a trace (one service call, one DB query)
Trace ID	Unique ID propagated through all HTTP headers for a request
Context propagation	Passing the trace ID in HTTP headers (traceparent, X-B3-TraceId) so each service can attach its span to the same trace
Parent span	The span that initiated a child operation

---

OpenTelemetry — The Standard

OpenTelemetry (OTel) is the CNCF standard for instrumentation — a vendor-neutral SDK and wire protocol that works with any tracing backend (Jaeger, Tempo, Zipkin, Datadog, Honeycomb).

flowchart TD

    App["Application <br/> (OTel SDK)"]
    App-->|"OTLP gRPC/HTTP"| Collector["OTel Collector <br/> (receive → process → export)"]

    Collector-->Tempo["Grafana Tempo <br/> (trace storage)"]
    Collector-->Prom["Prometheus <br/> (metrics)"]
    Collector-->Loki["Loki <br/> (logs)"]

OTel Collector

The OTel Collector is the central pipeline component —

• it receives telemetry from applications
• processes it (sampling, enrichment, batching)
• exports to one or more backends.

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
  resource:
    attributes:
    - action: insert
      key: cluster
      value: dgx-prod

exporters:
  otlp/tempo:
    endpoint: tempo.monitoring.svc.cluster.local:4317
    tls:
      insecure: true
  prometheusremotewrite:
    endpoint: http://prometheus.monitoring.svc.cluster.local:9090/api/v1/write
  loki:
    endpoint: http://loki.monitoring.svc.cluster.local:3100/loki/api/v1/push

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch, resource]
      exporters: [otlp/tempo]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheusremotewrite]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [loki]

OTel Operator — Auto-Instrumentation

The OpenTelemetry Operator can inject instrumentation into Pods automatically — no code changes:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: python-instrumentation
  namespace: inference
spec:
  exporter:
    endpoint: http://otel-collector.monitoring.svc.cluster.local:4318
  propagators:
    - tracecontext
    - baggage
  python:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latest

# Add annotation to Pod — OTel Operator injects the agent
metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-python: "true"

The operator mounts the OTel SDK into the Pod as an init container and sets PYTHONPATH — the application emits traces without any SDK calls in application code.

---

Tracing Backends

Grafana Tempo

Tempo is Grafana's trace backend — stores traces on object storage (S3/GCS), queryable via TraceQL:

helm upgrade --install tempo grafana/tempo-distributed \
  --namespace monitoring \
  --set storage.trace.backend=s3 \
  --set storage.trace.s3.bucket=my-traces-bucket

# TraceQL — find slow GPU inference spans
{ span.service.name = "nim-llm" && duration > 500ms }

Jaeger

Jaeger (CNCF graduated) is the original open-source distributed tracing platform:

helm upgrade --install jaeger jaegertracing/jaeger \
  --namespace monitoring \
  --set storage.type=elasticsearch \
  --set elasticsearch.host=elasticsearch.logging.svc.cluster.local

Jaeger and Tempo both accept the OTLP protocol from OTel — which backend you use is an operational choice, not a code change.

---

The Grafana Observability Stack (LGTM)

flowchart TB

subgraph Backends
  Loki["Loki <br/> (logs)"]
  Mimir["Mimir / Prometheus <br/> (metrics)"]
  Tempo["Tempo <br/> (traces)"]
end

subgraph Collection
  FluentBit["Fluent Bit <br/> (log collector)"]
  OTelCol["OTel Collector <br/> (metrics + traces)"]
  DCGM["DCGM Exporter <br/> (GPU metrics)"]
end

FluentBit --> Loki
OTelCol --> Mimir
OTelCol --> Tempo
DCGM --> Mimir

Loki & Mimir & Tempo --> Grafana["Grafana <br/> (unified dashboard)"]

The power of running all three backends behind Grafana: correlated signals.

A GPU temperature spike alert fires → you click the Grafana panel → jump to logs from that node at that timestamp → jump from a logged error to the trace showing which inference request caused the GPU to spike → see the full call chain. All without leaving Grafana.

---

Kubernetes-Native Metrics

Kubernetes itself exposes two metrics APIs:

API	Source	What it provides
metrics.k8s.io	Metrics Server	Current CPU/memory per Pod and Node (used by kubectl top and HPA)
custom.metrics.k8s.io	Prometheus Adapter	Custom metrics from Prometheus (used by HPA for GPU utilization)
external.metrics.k8s.io	KEDA	External system metrics (used by KEDA ScaledObjects)

# Uses Metrics Server
kubectl top nodes
kubectl top pods --all-namespaces --sort-by=cpu

# Both require Prometheus Adapter or KEDA
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1"

Metrics Server is lightweight — it only holds the current value, not history. Prometheus stores the full time-series history and is the right choice for dashboards and alerting.

---

GPU Observability on DGX

The GPU Operator deploys DCGM Exporter as a DaemonSet. Combined with kube-prometheus-stack, every GPU metric is automatically available in Prometheus and Grafana.

Key dashboards to build for a DGX cluster:

Dashboard	Key panels
GPU Fleet Health	Per-GPU utilization heatmap, ECC error count, temperature, power draw
Training Job Tracker	NVLink bandwidth (AllReduce throughput), GPU memory per worker, loss curve
Inference SLO	Request latency p50/p99, token throughput, GPU utilization per NIM Pod
Cluster Capacity	GPU allocated vs available by node group, Kueue queue depth

# NIM inference throughput (tokens per second)
rate(nim_token_output_total{namespace="inference"}[1m])

# Training AllReduce bottleneck — NVLink bandwidth approaching saturation
DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL / 900e9 * 100  # % of 900 GB/s NVSwitch bandwidth

# Kueue queue depth for alert
kueue_pending_workloads{cluster_queue="training-queue"} > 10

---

Key Takeaways

Cloud-native observability goes beyond simple monitoring by combining metrics, logs, and traces to provide deep insight into distributed systems.

• Prometheus collects metrics.
• Loki aggregates logs.
• Tempo stores distributed traces.
• OpenTelemetry standardizes instrumentation and telemetry collection.
• Grafana unifies visualization and analysis.
• AlertManager ensures the right teams are notified when systems deviate from expected behavior.

Together, these components enable engineers to detect issues quickly, trace requests across microservices, identify root causes, and maintain reliable cloud-native applications at scale.

Pillar	Tool	Stores	Queries with
Metrics	Prometheus	Numeric time-series	PromQL
Logs	Loki	Log streams (labels only indexed)	LogQL
Traces	Tempo / Jaeger	Span trees	TraceQL / Jaeger UI
Visualization	Grafana	Nothing (reads from above)	Dashboard panels
Collection	Fluent Bit	Nothing (ships logs to Loki)	—
Collection	OTel Collector	Nothing (pipelines to backends)	—
Instrumentation	OTel SDK / Operator	Nothing (emits signals)	OTLP protocol
Alerting	Alertmanager	Nothing (routes Prometheus alerts)	Routing rules

Observability is not monitoring. Monitoring tells you when something is broken.

Observability tells you why. The difference is whether you can ask a new question — one you didn't think to instrument for — and get an answer from the data you already have.

For a GPU cluster, that means correlating an AllReduce slowdown (NVLink bandwidth metric) with a noisy-neighbor Pod (logs) to a specific training run (trace) without guessing which component to look at first.

---

Related Posts

• GPU Scheduling in Kubernetes — DCGM Exporter deployment by the GPU Operator; the source of the GPU metrics consumed by Prometheus in this post
• GPU Autoscaling: KEDA, HPA, Cluster Autoscaler — how DCGM metrics flow from Prometheus through KEDA or the Prometheus Adapter to drive HPA scaling decisions
• Kubernetes Performance at Scale — the Kubernetes control plane metrics (API Server latency, etcd WAL fsync, scheduler throughput) that Prometheus monitors
• AI Infra Computing: GPU and DGX Systems — the hardware whose health DCGM monitors: H100 SM utilization, NVLink bandwidth, HBM memory