Kubernetes Pod Internals: What a Pod Really Is

A pod is Kubernetes' atomic scheduling unit. But that definition is Kubernetes-speak. From the operating system's point of view, a pod is a carefully constructed set of Linux primitives — namespaces, cgroups, and a ghost process called pause. Understanding both views is what separates someone who uses Kubernetes from someone who can debug it.

---

The Kubernetes View

From Kubernetes, a pod is:

• The smallest deployable unit — you never schedule a container directly, always a pod
• A group of one or more containers that share lifecycle, network, and storage
• An atomic scheduling decision — all containers in a pod land on the same node
• A temporary entity — pods are mortal; controllers (Deployments, StatefulSets) recreate them

apiVersion: v1
kind: Pod
metadata:
  name: web-pod
spec:
  containers:
  - name: nginx
    image: nginx:1.25
    ports:
    - containerPort: 80
  - name: log-shipper          # sidecar — same pod, same node
    image: fluent/fluent-bit

These two containers:

• Share the same IP address
• Can reach each other on localhost
• Share the same volume mounts if declared
• Start and stop together

---

The OS View

From Linux, there is no such thing as a pod. The OS only understands processes, namespaces, and cgroups. A pod is implemented as a group of processes sharing a set of Linux namespaces.

Linux Namespaces — The Isolation Primitives

A namespace isolates a specific aspect of the system. Kubernetes uses six of them:

Namespace	Isolates	Shared in a pod?
NET	Network interfaces, routes, iptables	✅ Yes — all containers share one IP
UTS	Hostname and domain name	✅ Yes — pod has one hostname
IPC	SysV IPC, POSIX message queues	✅ Yes — containers can use shared memory
PID	Process ID tree	⚠️ Optional (shareProcessNamespace: true)
MNT	Filesystem mount points	❌ No — each container has its own
USER	UID/GID mapping	❌ No — each container has its own

The NET namespace being shared is what makes localhost work between containers in the same pod — they literally share the same network interface.

cgroups — The Resource Limits

Linux cgroups enforce the resources.requests and resources.limits you declare. Each pod gets its own cgroup hierarchy:

/sys/fs/cgroup/kubepods/
  burstable/
    pod<uid>/               ← one cgroup per pod
      cpu.shares            ← from requests.cpu
      memory.limit_in_bytes ← from limits.memory
      <container-id>/       ← one sub-cgroup per container
        cpu.cfs_quota_us    ← from limits.cpu

# See the cgroup for a running pod
cat /proc/$(pgrep nginx)/cgroup
# 11:memory:/kubepods/burstable/pod<uid>/<container-id>

# See current memory limit enforced by cgroup
cat /sys/fs/cgroup/memory/kubepods/burstable/pod<uid>/memory.limit_in_bytes

---

The Pause Container — The Pod's Skeleton

Here is the part most Kubernetes users never see.

Every pod contains a hidden container called pause (also called the infra container or sandbox). You never declare it — Kubernetes injects it automatically.

# On a node, list ALL containers including pause
crictl ps | grep pause
# pause  k8s.gcr.io/pause:3.9  ...  web-pod
# pause  k8s.gcr.io/pause:3.9  ...  gpu-job

Why does the pause container exist?

When containers start and stop inside a pod, the Linux namespaces they share would normally be destroyed when the last process using them exits. The pause container is a minimal process (about 700KB) that does nothing except:

1. Hold the network namespace open — so the pod's IP address survives container restarts
2. Hold the IPC namespace open — so shared memory segments persist
3. Act as PID 1 — reaping zombie child processes

Pod "web-pod"
│
├── pause (PID 1)          ← creates NET, UTS, IPC namespaces. Never exits.
│     └── holds: NET namespace (IP: 10.0.0.42)
│
├── nginx                  ← joins pause's NET namespace. Shares 10.0.0.42.
│     └── pid: 42
│
└── fluent-bit             ← joins pause's NET namespace. Shares 10.0.0.42.
      └── pid: 87

If nginx crashes and restarts, its new process re-joins the existing NET namespace from pause. The pod's IP never changes. Without pause, the IP would be destroyed and recreated on every container restart.

---

What Happens Between kubectl apply and Your Process Running

This is the full chain for a new pod:

kubectl apply -f pod.yaml
       │
       ▼
1. API Server validates and persists pod spec to etcd
       │
       ▼
2. Scheduler watches for unscheduled pods
   → Filters nodes (resource fit, taints, affinity)
   → Scores nodes
   → Binds pod to node (writes nodeName to etcd)
       │
       ▼
3. Kubelet on target node watches API Server
   → Detects pod bound to its node
       │
       ▼
4. Kubelet calls Container Runtime Interface (CRI)
   → Runtime (containerd) creates pod sandbox
   → Pulls pause image
   → Starts pause container → creates NET, UTS, IPC namespaces
       │
       ▼
5. Init containers run sequentially (if any)
   → Each must exit 0 before next starts
   → All join the pause namespaces
       │
       ▼
6. Main containers start in parallel
   → Each joins pause's NET, UTS, IPC namespaces
   → Each gets its own MNT namespace
   → cgroups enforced per container
   → nvidia-container-toolkit injects GPU devices (if requested)
       │
       ▼
7. Kubelet runs probes
   → startupProbe: gates readiness/liveness until app boots
   → readinessProbe: gates Service traffic
   → livenessProbe: restarts container on failure
       │
       ▼
8. Pod status = Running, Conditions: Ready=True
   → Service endpoints updated
   → Traffic routed to pod IP

---

Pod Lifecycle Phases

Pending → Running → Succeeded
                 ↘ Failed
                 ↘ Unknown

Phase	Meaning
Pending	Accepted by API server, waiting to be scheduled or for images to pull
Running	Bound to a node, at least one container running
Succeeded	All containers exited 0, restartPolicy: Never/OnFailure
Failed	At least one container exited non-zero, not going to restart
Unknown	Node communication lost — kubelet not responding

kubectl get pod web-pod -o jsonpath='{.status.phase}'

# Detailed conditions
kubectl get pod web-pod -o jsonpath='{.status.conditions[*]}'
# PodScheduled=True, Initialized=True, ContainersReady=True, Ready=True

---

Init Containers

Init containers run before any main container starts. They share the same volumes but run in sequence, each must exit 0.

spec:
  initContainers:
  - name: wait-for-db
    image: busybox
    command: ['sh', '-c', 'until nc -z postgres 5432; do sleep 2; done']

  - name: migrate
    image: myapp:latest
    command: ['python', 'manage.py', 'migrate']

  containers:
  - name: app
    image: myapp:latest     # only starts after both inits succeed

Use cases:

• Wait for a dependency (database, service) to be ready
• Run DB migrations before the app starts
• Pre-populate a shared volume (download model weights, config)
• Set file permissions on a shared volume

---

Sidecar Containers (Kubernetes 1.29+)

Native sidecar containers are a new type that start before main containers (like init containers) but keep running alongside them (like regular containers). They also survive if the main containers restart.

spec:
  initContainers:
  - name: log-collector
    image: fluent/fluent-bit
    restartPolicy: Always      # this is what makes it a sidecar
  containers:
  - name: app
    image: myapp:latest

Before native sidecars, you'd put log shippers in containers: — but they had no guaranteed start order. Native sidecars solve that.

---

Shared Process Namespace

By default, each container has its own PID namespace — container A cannot see container B's processes.

Enable shareProcessNamespace to let containers see each other's processes (useful for debugging sidecars):

spec:
  shareProcessNamespace: true
  containers:
  - name: app
    image: myapp:latest
  - name: debugger
    image: nicolaka/netshoot
    # Can now run: ps aux  → sees app's processes
    # Can now run: kill -USR1 <app-pid>  → sends signal to app

---

Pod Conditions and Readiness Gates

A pod is only added to Service endpoints when Ready=True. You can add custom readiness gates — extra conditions that must be true before the pod is considered ready:

spec:
  readinessGates:
  - conditionType: "feature-flags/ready"   # custom condition
  containers:
  - name: app
    image: myapp:latest

Your application (or an external controller) must then patch the pod status:

kubectl patch pod web-pod --subresource=status --type=merge \
  -p '{"status":{"conditions":[{"type":"feature-flags/ready","status":"True"}]}}'

Used for: waiting for feature flag sync, waiting for sidecar warm-up, external health checks.

---

Lifecycle Hooks

Containers can run code at two lifecycle events:

containers:
- name: app
  lifecycle:
    postStart:            # runs immediately after container starts
      exec:
        command: ["/bin/sh", "-c", "echo started > /tmp/started"]
    preStop:              # runs before container receives SIGTERM
      exec:
        command: ["/bin/sh", "-c", "nginx -s quit; sleep 5"]

Hook	When	Use for
postStart	Immediately after container starts (async — may run before ENTRYPOINT)	Bootstrap tasks, registration
preStop	Before SIGTERM is sent	Graceful shutdown, deregistration, drain

preStop is critical for zero-downtime deployments. Without it, the pod may be removed from the Service endpoint while still processing in-flight requests.

---

What kubectl exec Actually Does

kubectl exec -it web-pod -- bash

Under the hood:

1. kubectl sends exec request to API Server
2. API Server proxies to kubelet on the node
3. Kubelet calls exec on the container runtime
4. Container runtime uses nsenter to enter the container's namespaces
5. A new bash process is spawned inside those namespaces

The new process shares the same NET, UTS, IPC namespaces — but it is not a child of the container's PID 1. It's a sibling process that joined the namespaces.

# What kubectl exec actually runs under the hood:
nsenter --target <container-pid> \
        --mount --uts --ipc --net \
        -- bash

---

🧠 Interview Questions

Q: Two containers in the same pod — can they communicate on localhost? Yes. They share the NET namespace via the pause container. Both bind to the same loopback interface at 127.0.0.1.

Q: If the main container in a pod crashes, what happens to the pod IP? Nothing — the IP is held by the pause container which keeps running. The main container restarts and re-joins the existing NET namespace. The IP is preserved.

Q: What is the difference between a pod restart and a pod recreation? A restart keeps the same pod object, same IP, same UID — the container process exits and kubelet starts it again per restartPolicy. A recreation (by a Deployment rolling update or node failure) creates a brand-new pod object with a new IP and new UID.

Q: Can two containers in the same pod share memory (shared memory segment)? Yes — they share the IPC namespace. A POSIX shared memory segment created by container A is visible to container B at /dev/shm. This is used by some ML frameworks for fast intra-pod IPC.

Q: What happens during a pod's graceful termination?

kubectl delete pod web-pod
       │
       ▼
1. Pod marked for deletion (deletionTimestamp set)
2. Pod removed from Service endpoints immediately
3. preStop hook runs (if defined)
4. SIGTERM sent to container PID 1
5. Grace period countdown starts (default 30s)
6. If container still running after grace period → SIGKILL
7. Pod object deleted from etcd

---

Related Posts

• Kubernetes Scheduler Internals — how the scheduler picks which node the pod lands on
• Kubernetes Resource Allocation — how requests, limits, and QoS classes work at the cgroup level
• Kubernetes Networking — how the pod IP is assigned and how traffic reaches it
• GPU Scheduling in Kubernetes — how nvidia-container-toolkit injects GPU devices into the pod's namespaces
• Kubernetes Controllers and Informers — how Deployments and StatefulSets manage pod lifecycle