Storage

AI workloads require:

• Extremely high throughput
• Parallel access from many GPUs
• Low latency during training
• Scalable capacity for datasets and checkpoints

Bottlenecks in AI Storage

Key Principle: GPUs must never sit idle waiting for data.

Common issues:

• Insufficient I/O throughput
• Network congestion
• Poor file system scaling
• CPU bottlenecks during data movement

Impact: GPU underutilization.

---

Tiered Storage Architecture

AI data centers use a hybrid storage model.

1. 🔥 Hot Tier (Fastest)

1.1 NVMe SSD (Local Storage)

• Directly attached to server
• Very high IOPS and throughput
• Used for:
  • Active model training
  • Temporary datasets
  • Checkpoints

Limited capacity but extremely fast.

1.2. Network File Systems (Shared Storage)

• Accessible by multiple nodes
• Moderate latency but most common for shared access
• Stored as Blocks or Files
• Used for:
  • Shared datasets
  • Model checkpoints er (Shared High-Speed)

1.3. Parallel & Distributed File Systems

• Shared across cluster
• High bandwidth
• Scales horizontally
• Supports many GPUs simultaneously

Used for:

• Distributed training
• Shared datasets
• Large-scale HPC workloads

2. ❄ Cold Tier (Long-Term Storage)

2.1 Object Storage

• Massive scalability
• Lower cost per TB
• Higher latency

Used for:

• Raw datasets
• Archived models
• Logs
• Historical checkpoints

Examples:

• S3-compatible systems
• Cloud object storage

Data Locality

Better performance when:

• Data is closer to compute
• Fewer network hops required

Hierarchy: Local NVMe > Parallel FS > Object Storage

---

Storage Access Patterns in AI

During Training

• Large sequential reads
• Multi-node concurrent access
• Frequent checkpoint writes

Requires:

• High throughput
• Parallel file systems
• RDMA support

During Inference

• Smaller model loads
• Lower bandwidth needs
• Latency-sensitive

Often served via:

• NVMe
• Optimized storage pipelines

---

RDMA & Storage

Traditional storage path: Storage → CPU → System Memory → GPU

With acceleration: Storage → GPU Memory (Direct)

GPUDirect Storage

• Bypasses CPU and system memory
• Direct path from NVMe or parallel storage to GPU
• Reduces bottlenecks
• Improves training speed

Best for:

• Large dataset ingestion
• High-performance training clusters

NVMe over Fabrics (NVMe-oF)

• Extends NVMe across network
• Enables remote high-speed storage access
• Often combined with RDMA
• Used in HPC and AI clusters

Storage Networking Considerations

Storage traffic must be:

• Isolated from compute fabric
• High bandwidth
• Low contention
• Predictable

AI clusters often separate:

• Compute traffic
• Storage traffic
• Management traffic

---

Storage Scalability

AI datasets grow rapidly.

Storage must:

• Scale capacity easily
• Maintain performance at scale
• Support multi-node access

Parallel file systems scale horizontally by:

• Adding storage nodes
• Distributing metadata

---

Storage and Checkpointing

During training:

• Models save checkpoints periodically
• Checkpoints can be large (GBs–TBs)

Storage must handle:

• Frequent writes
• Multi-GPU simultaneous checkpoints
• Recovery after failure

---

RAID & Data Protection

RAID used for:

• Redundancy
• Performance improvement
• Fault tolerance

In large AI systems:

• Erasure coding often used
• Object storage provides durability

---

Storage in Cloud vs On-Prem

Cloud

• Object storage dominant
• Elastic scaling
• Pay-as-you-go

On-Prem

• Full control
• Parallel file systems common
• Lower long-term cost at scale

---

Exam Scenarios to Recognize

If question mentions:

• GPUs starving for data → Storage bottleneck
• Massive shared dataset across nodes → Parallel file system
• Long-term archive → Object storage
• Direct storage-to-GPU transfer → GPUDirect Storage
• Ultra-fast local I/O → NVMe SSD

Quick Memory Anchors

• NVMe = Fastest local storage
• Parallel FS = Shared high-speed cluster storage
• Object storage = Massive, cheap, long-term
• GPUDirect Storage = Bypass CPU
• Training = High throughput demand
• Inference = Lower bandwidth, latency focus