Image Internals: From OCI Layers to a Running Container

When you run kubectl apply, a YAML file becomes a running process on a node. That process started from bytes stored in a registry. Understanding every step between those two points — image pull, layer assembly, snapshot creation, pod sandbox, namespace join — is what separates someone who uses containers from someone who can debug them at any layer of the stack.

---

The Big Picture

Registry (image bytes)
       │
       │ 1. Manifest fetch + layer pull
       ▼
containerd image store
       │
       │ 2. Snapshot (OverlayFS assembly)
       ▼
OCI bundle (rootfs + config.json)
       │
       │ 3. Pod sandbox (pause container)
       ▼
Network namespace + IP assigned
       │
       │ 4. Container joins sandbox
       ▼
runc creates namespaces, cgroups, pivot_root
       │
       │ 5. exec()
       ▼
Your process running as PID 1 inside the container

---

Step 1: What an Image Actually Is

An OCI image is not a single file. It is three things stored in a content-addressable store:

The Manifest

The manifest is the index card — it says what layers make up the image and where to find the config.

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:a1b2c3...",
    "size": 7023
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:d4e5f6...",
      "size": 31379766
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:a7b8c9...",
      "size": 25165824
    }
  ]
}

The Config

The config is the runtime metadata — what command to run, what env vars to set, what the working directory is, what user to run as:

{
  "architecture": "amd64",
  "os": "linux",
  "config": {
    "Env": ["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"],
    "Cmd": ["nginx", "-g", "daemon off;"],
    "WorkingDir": "/",
    "User": "",
    "ExposedPorts": {"80/tcp": {}}
  },
  "rootfs": {
    "type": "layers",
    "diff_ids": [
      "sha256:layer1-uncompressed-digest...",
      "sha256:layer2-uncompressed-digest..."
    ]
  },
  "history": [...]
}

The Layers

Each layer is a gzip-compressed tar archive containing the filesystem diff for that build step. A layer can add files (new path), modify files (whiteout + new version), or delete files (.wh. whiteout marker).

# A whiteout file tells OverlayFS to hide the file from lower layers
.wh.etc/passwd        # deletes etc/passwd from the merged view
.wh..wh..opq          # opaque whiteout — hides entire directory from lower layers

---

Step 2: Content-Addressable Storage

Every layer, config, and manifest is identified by its SHA-256 digest. The digest is a hash of the content itself — change one byte, get a completely different digest.

This means:

Tag → Manifest digest → Config digest + Layer digests

Tags are mutable pointers. Digests are immutable.

# Pull by digest — always the exact same bytes
docker pull nginx@sha256:1234abcd...

# Pull by tag — could be different bytes tomorrow
docker pull nginx:1.25

# See the digest of a local image
docker images --digests nginx
# nginx  1.25  sha256:1234abcd...  2 weeks ago  187MB

# On disk, containerd stores everything under its content store
ls /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/
# d4e5f6...  ← layer 1 compressed tarball
# a7b8c9...  ← layer 2 compressed tarball
# a1b2c3...  ← config JSON

Two images sharing a base layer reference the exact same bytes on disk — no duplication.

---

Step 3: Registry Pull Flow

When a node needs an image it doesn't have, kubelet tells containerd to pull it:

Kubelet
  │  CRI: PullImage(nginx:1.25)
  ▼
containerd
  │
  ├─ 1. Resolve tag → registry API: GET /v2/nginx/manifests/1.25
  │     Headers: Authorization: Bearer <token>
  │     Response: manifest JSON + digest header
  │
  ├─ 2. Check local content store for each layer digest
  │     → Layer already present? Skip. Missing? Download.
  │
  ├─ 3. For each missing layer:
  │     GET /v2/nginx/blobs/sha256:<digest>
  │     Stream to /var/lib/containerd/.../ingest/<random>
  │     Verify: sha256(downloaded bytes) == digest from manifest
  │     Move to content store on success
  │
  └─ 4. Unpack config JSON, store in metadata DB (bbolt)

# Watch a pull happen with verbose output
containerd --log-level debug &
ctr images pull docker.io/library/nginx:1.25

# Or via crictl on a Kubernetes node
crictl pull nginx:1.25

Layer Deduplication at Pull Time

# Pull nginx — downloads 3 layers
ctr images pull nginx:1.25
# layer 1: sha256:abc...  31MB  pulled
# layer 2: sha256:def...  25MB  pulled
# layer 3: sha256:ghi...  1MB   pulled

# Pull nginx patch version — only pulls the changed layer
ctr images pull nginx:1.26
# layer 1: sha256:abc...  31MB  exists ← skipped, same bytes
# layer 2: sha256:def...  25MB  exists ← skipped
# layer 3: sha256:xyz...  1MB   pulled ← only this changed

---

Step 4: Snapshots — Assembling the Filesystem

containerd does not directly mount OverlayFS. It uses a snapshotter — a pluggable interface for managing the filesystem state of a container.

The default snapshotter on Linux is overlayfs.

Snapshot Chain

Each image layer becomes a snapshot — an immutable point-in-time view of the filesystem up to that layer:

Snapshot 0: ubuntu base layer     ← lowerdir[2]
     │
     ▼
Snapshot 1: + python installed    ← lowerdir[1]
     │
     ▼
Snapshot 2: + app code copied     ← lowerdir[0]
     │
     ▼
Active Snapshot (container)       ← upperdir (read-write)

# See snapshots on a node
ctr snapshots ls
# KEY                               PARENT          KIND
# sha256:abc...                                     Committed   ← layer 0
# sha256:def...        sha256:abc...                Committed   ← layer 1
# sha256:ghi...        sha256:def...                Committed   ← layer 2
# my-container-active  sha256:ghi...                Active      ← rw layer

The OverlayFS Mount

When a container starts, containerd assembles the snapshot chain into an OverlayFS mount:

mount -t overlay overlay \
  -o lowerdir=/snapshots/ghi/fs:/snapshots/def/fs:/snapshots/abc/fs,\
     upperdir=/snapshots/active/fs,\
     workdir=/snapshots/active/work \
  /run/containerd/io.containerd.runtime.v2.task/<id>/rootfs

What the container process sees at /
│
├── /bin, /lib, /usr          ← from ubuntu layer (lowerdir[2])
├── /usr/local/lib/python3.11 ← from python layer (lowerdir[1])
├── /app/                     ← from app layer (lowerdir[0])
└── /tmp/, /var/log/          ← writes go here (upperdir), ephemeral

Every read from a lower layer is zero-copy — the kernel serves it directly from the read-only snapshot. Every write triggers a copy-up to upperdir first.

---

Step 5: OCI Bundle — The Runtime's Input

Before calling runc, containerd produces an OCI bundle: a directory with two things:

/run/containerd/io.containerd.runtime.v2.task/<pod>/<container>/
├── rootfs/          ← the OverlayFS merged mount (the container's filesystem)
└── config.json      ← OCI Runtime Spec (namespaces, cgroups, mounts, process)

The config.json is generated by containerd from:

• The image config (env, cmd, user, working dir)
• The pod spec (resource limits, security context, volume mounts)
• The pod sandbox namespaces (NET, IPC, UTS to join)

{
  "ociVersion": "1.0.2",
  "process": {
    "user": {"uid": 1000, "gid": 1000},
    "args": ["nginx", "-g", "daemon off;"],
    "env": ["PATH=/usr/local/sbin:..."],
    "cwd": "/"
  },
  "root": {"path": "rootfs", "readonly": false},
  "namespaces": [
    {"type": "pid"},
    {"type": "mount"},
    {"type": "network", "path": "/proc/12345/ns/net"}  ← join pause container's NET ns
  ],
  "linux": {
    "resources": {
      "memory": {"limit": 536870912},
      "cpu": {"quota": 200000, "period": 100000}
    },
    "seccomp": {...},
    "maskedPaths": ["/proc/acpi", "/sys/firmware"]
  }
}

---

Step 6: Pod Sandbox — The Pause Container

A pod is not a container. A pod is a shared execution environment — a set of namespaces that multiple containers all join. The holder of those namespaces is the pause container.

# See the pause container on any node
crictl ps | grep pause
# <id>   registry.k8s.io/pause:3.9   12h   Running   POD  my-pod

What the Pause Container Does

kubelet creates pod sandbox
       │
       ▼
containerd pulls pause:3.9 (if not cached)
       │
       ▼
runc starts pause container with:
  - New NET namespace  → gets veth pair, assigned pod IP by CNI
  - New IPC namespace  → SysV IPC and POSIX shared memory scope
  - New UTS namespace  → pod hostname
       │
       ▼
pause process runs: for(;;) pause();   ← does nothing, holds namespaces open
       │
       ▼
CNI plugin wires pod IP into the NET namespace

The pause container runs an infinite loop doing nothing (pause() syscall). Its only job is to keep the namespaces alive. If an app container crashes and restarts, the pod IP is preserved because the NET namespace belongs to the pause container, not to the app container.

# Confirm: pause container holds the NET namespace
PAUSE_PID=$(crictl inspect <pause-id> | jq -r '.info.pid')
ls -la /proc/$PAUSE_PID/ns/net
# lrwxrwxrwx ... net -> net:[4026532300]

APP_PID=$(crictl inspect <app-id> | jq -r '.info.pid')
ls -la /proc/$APP_PID/ns/net
# lrwxrwxrwx ... net -> net:[4026532300]   ← same inode — same namespace

---

Step 7: runc Creates the Container

containerd hands the OCI bundle to containerd-shim-runc-v2, which calls runc:

runc create <container-id> --bundle /path/to/bundle
       │
       ├─ 1. Read config.json
       │
       ├─ 2. clone() — create new process in new namespaces
       │     CLONE_NEWPID  → new PID namespace (container gets PID 1)
       │     CLONE_NEWNS   → new MNT namespace
       │     (NET/IPC/UTS: join pause container's ns via "path" in config.json)
       │
       ├─ 3. Set up cgroup — write limits to /sys/fs/cgroup/...
       │     memory.max  = limits.memory
       │     cpu.max     = limits.cpu quota/period
       │
       ├─ 4. pivot_root — make the OverlayFS rootfs the new / inside MNT ns
       │
       ├─ 5. Mount special filesystems inside the new root:
       │     /proc  → new procfs (scoped to container's PID namespace)
       │     /sys   → sysfs (partially masked)
       │     /dev   → devtmpfs (minimal device set)
       │     /etc/resolv.conf → bind mount from kubelet
       │     /var/run/secrets/kubernetes.io/serviceaccount → projected volume
       │
       ├─ 6. Drop capabilities to configured set
       │     Apply seccomp filter
       │     Apply AppArmor profile
       │
       └─ 7. exec() → replace runc with nginx process
              Now: nginx is PID 1 inside its own PID namespace
              Sees: only OverlayFS filesystem, only pod's network, only its cgroup

---

Step 8: End-to-End — kubectl apply to Running Process

User: kubectl apply -f deployment.yaml
         │
         ▼
1.  API Server validates and writes Pod spec to etcd

2.  Scheduler watches for unscheduled pods
    → Runs filter + score → assigns node
    → Writes pod.spec.nodeName to etcd

3.  Kubelet on that node watches for pods assigned to it
    → Admits the pod (resource fits allocatable)

4.  Kubelet calls containerd CRI: RunPodSandbox()
    → containerd starts pause container
    → CNI plugin creates veth pair, assigns pod IP

5.  Kubelet calls containerd CRI: PullImage() for each container
    → containerd checks local content store
    → Downloads missing layers, verifies digests
    → Creates snapshot chain (OverlayFS)

6.  Kubelet calls containerd CRI: CreateContainer()
    → containerd generates OCI config.json
    → config.json references pause container namespaces

7.  Kubelet calls containerd CRI: StartContainer()
    → containerd-shim calls runc
    → runc: clone(), cgroup setup, pivot_root, exec()
    → nginx starts as PID 1 inside the container

8.  Kubelet runs liveness/readiness probes
    → On success: sets pod.status = Running
    → API Server writes status to etcd
    → kubectl get pods shows Running

---

Image Layer to Filesystem: What You Actually See

# Inspect the layer structure of a local image
docker inspect python:3.11-slim | jq '.[0].RootFS.Layers'
# [
#   "sha256:layer1...",   ← debian:bookworm-slim base
#   "sha256:layer2...",   ← apt-get install python3
#   "sha256:layer3...",   ← pip defaults
#   "sha256:layer4..."    ← python runtime setup
# ]

# Walk the actual OverlayFS mount for a running container
CONTAINER_ID=$(docker ps -q --filter name=myapp)
OVERLAY=$(docker inspect $CONTAINER_ID | jq -r '.[0].GraphDriver.Data.MergedDir')
ls $OVERLAY
# bin  dev  etc  home  lib  proc  root  sys  tmp  usr  var  app

# See what a container has written to its rw layer
UPPER=$(docker inspect $CONTAINER_ID | jq -r '.[0].GraphDriver.Data.UpperDir')
find $UPPER -type f 2>/dev/null
# ./var/log/nginx/access.log    ← nginx wrote this
# ./tmp/somefile                ← app wrote this
# → both disappear on container removal

---

Image Size vs Runtime Size

Metric	Meaning	How to check
Image size	Sum of all compressed layer tarballs	docker images nginx
Virtual size	Sum of all uncompressed layers (what OverlayFS exposes)	docker images --format "{{.VirtualSize}}"
Container size	Bytes written to upperdir (rw layer only)	docker ps -s
Shared layers	Bytes shared with other containers (from lowerdir)	docker system df -v

# How much disk is really being used?
docker system df
# TYPE            TOTAL    ACTIVE   SIZE      RECLAIMABLE
# Images          12       3        4.2GB     2.8GB (66% reclaimable)
# Containers      3        3        128MB     0B
# Local Volumes   5        2        9.1GB     5.2GB

# Remove dangling layers (untagged, unreferenced)
docker image prune

---

Linux Concepts: Image to Container

Stage	Linux Mechanism	What Happens
Layer storage	Content-addressable filesystem, SHA-256	Each layer stored once, identified by digest
Layer download	HTTP chunked transfer, splice(2)	Stream from registry to disk, verify digest
Layer extraction	tar + gzip, write(2)	Decompress tarball into snapshot directory
Snapshot chain	Hardlinks + directory tree	Each layer's snapshot points to parent
Filesystem assembly	OverlayFS mount()	Stacks snapshots into unified rootfs view
Copy-on-Write	Page fault → copy-up	Modified file copied from lowerdir to upperdir
Whiteout	Special .wh. files	OverlayFS hides deleted files from lower layers
Namespace creation	`clone(CLONE_NEWPID	CLONE_NEWNS)`
Namespace joining	setns(fd) with /proc/<pid>/ns/net	Container joins pause container's NET namespace
Root change	pivot_root(new_root, put_old)	Container's / becomes the OverlayFS merged dir
Resource limits	Write to /sys/fs/cgroup/	CPU quota and memory max applied before exec
Process start	exec(path, argv, envp)	runc replaced by container binary, becomes PID 1
Pod IP	Veth pair + bridge/VXLAN + CNI	pause container NET ns gets an IP from the CNI plugin

---

🧠 Interview Questions

Q: Two containers in the same pod share a network. How is this implemented at the kernel level? Both containers have a "network" entry in their OCI config.json pointing to /proc/<pause-pid>/ns/net. When runc calls setns(fd, CLONE_NEWNET) for each app container, they all join the same network namespace inode. From the kernel's perspective, they are the same network stack — same eth0, same IP, same routing table, same loopback.

Q: If I delete a file inside a container, does it free up disk space? No — if the file was in a lower image layer. Deleting it writes a .wh.<filename> whiteout marker to the upperdir. The original bytes in the lowerdir are still on disk. Disk is only freed when the image itself is removed (docker rmi). The whiteout actually costs a tiny amount of space.

Q: I have 20 containers all running the same nginx:1.25 image. How much disk do the image layers use? Exactly the same as one container — the layers are shared. All 20 containers share the same lowerdir snapshots. Each container has its own upperdir (written bytes only). docker system df shows shared vs unique bytes.

Q: A container restarts due to OOMKill. Does it get a new IP? No. The IP belongs to the pause container's NET namespace, which was not restarted. The app container process died, runc re-creates it, and it rejoins the same NET namespace via setns(). The pod IP is stable across container restarts — only pod deletion and recreation triggers a new IP.

Q: What is the difference between docker commit and a Dockerfile layer? Both produce a new image layer. docker commit snapshots the container's current upperdir into a new read-only layer appended to the image. A Dockerfile RUN instruction does the same thing automatically for each build step. The result is identical OCI layers — docker commit is just manual layer creation. Both approaches produce layers with the same OverlayFS representation.

Q: How does Kubernetes know if an image pull failed vs a container crash? Image pull failures surface as ErrImagePull → ImagePullBackOff in pod.status.containerStatuses[].state.waiting.reason. Container crashes appear as CrashLoopBackOff in the same field with exitCode populated. The distinction: image pull is a CRI PullImage failure; crash is a post-StartContainer process exit. kubectl describe pod shows Events with the exact error at each stage.

---

Related Posts

• Container Internals — Linux namespaces, cgroups, OverlayFS, and OCI spec in detail
• Kubernetes Pod Internals — pod lifecycle, pause container, and init containers
• Kubernetes Resource Allocation — how requests and limits map to cgroup values
• GPU Scheduling in Kubernetes — how nvidia-container-toolkit injects GPU devices at container start