Kubernetes Init:CrashLoopBackOff and Stuck Init Containers¶
Problem¶
A pod never gets past its init containers. The application container doesn't start because Kubernetes runs init containers first, one at a time, and each must finish successfully before the next begins.
NAME READY STATUS RESTARTS AGE
orders-api-7c9d8b6f5-k2x4p 0/1 Init:CrashLoopBackOff 5 (90s ago) 8m
orders-api-7c9d8b6f5-m8q2z 0/1 Init:0/2 0 15m
Symptoms¶
| Status | Meaning |
|---|---|
Init:0/2 for a long time |
The first of two init containers is still running — usually waiting forever |
Init:Error |
An init container exited non-zero |
Init:CrashLoopBackOff |
An init container keeps failing and the kubelet is backing off |
Init:ImagePullBackOff |
An init container's image can't be pulled — see ImagePullBackOff |
PodInitializing |
Init containers finished; app containers are starting |
kubectl logs <pod> for the main container returns nothing useful — it hasn't started. The clues are in the init container's logs.
Likely Causes¶
- A wait-for-dependency loop with no timeout. The init container polls a database or service that isn't reachable (wrong hostname, wrong namespace, network policy, dependency down).
- A database migration fails. The migration errors out, or conflicts with another replica running the same migration at the same time.
- A setup step lacks permissions.
chownor writes to a volume fail under a non-root security context. - Missing configuration. The init container needs a Secret or ConfigMap key that doesn't exist.
- Wrong image or command for the init container itself.
Diagnostic Commands¶
# 1. Which init container is running or failing, and its state
kubectl get pod orders-api-7c9d8b6f5-k2x4p \
-o jsonpath='{range .status.initContainerStatuses[*]}{.name}{"\t"}{.ready}{"\t"}{.restartCount}{"\t"}{.state}{"\n"}{end}'
# 2. Logs from the init container (use --previous if it's restarting)
kubectl logs orders-api-7c9d8b6f5-k2x4p -c wait-for-db
kubectl logs orders-api-7c9d8b6f5-k2x4p -c migrate --previous
# 3. Init container spec, events, and mounts
kubectl describe pod orders-api-7c9d8b6f5-k2x4p
kubectl get pod orders-api-7c9d8b6f5-k2x4p -o jsonpath='{range .spec.initContainers[*]}{.name}{"\t"}{.image}{"\t"}{.command}{"\n"}{end}'
To test the dependency the init container is waiting for, run a throwaway pod in the same namespace:
kubectl run nettest -n orders --rm -it --image=busybox:1.37 -- sh
# inside
nslookup orders-db.orders.svc.cluster.local
nc -zv -w 3 orders-db.orders.svc.cluster.local 5432
Step-by-Step Debugging¶
- Identify the init container that's blocking. Init containers run in order, so the first one that isn't
ready: trueis the culprit. - Read its logs, including
--previousfor restarting init containers. - If it's waiting, check the exact host and port it's polling, and test them from the same namespace. A Service in another namespace needs its full name:
orders-db.database.svc.cluster.local. - Check network policies in both namespaces if DNS resolves but connections time out — see Networking and Service Problems.
- If it's a migration, read the migration tool's error and check whether several replicas started migrating at the same moment.
- If it's a permission error, compare the init container's
securityContextwith the ownership of the volume it writes.
Solutions¶
Bound every wait¶
A wait that never gives up hangs rollouts silently. Fail after a reasonable time so the pod reports an error you can see:
initContainers:
- name: wait-for-db
image: busybox:1.37
command:
- sh
- -c
- |
for i in $(seq 1 60); do
nc -z -w 2 orders-db.database.svc.cluster.local 5432 && exit 0
echo "waiting for database ($i/60)"; sleep 5
done
echo "database not reachable after 5 minutes" >&2
exit 1
Better still, make the application retry its connections on startup, and drop the wait container entirely. Readiness probes then keep traffic away until it connects.
Run migrations once, not per replica¶
Running migrations in an init container means every replica — and every restart — tries to migrate. Move them to a dedicated Job that runs before the rollout, for example a Helm pre-upgrade hook or a pipeline step:
apiVersion: batch/v1
kind: Job
metadata:
name: orders-migrate-2-14-0
spec:
backoffLimit: 2
activeDeadlineSeconds: 600
template:
spec:
restartPolicy: Never
containers:
- name: migrate
image: registry.example.com/orders-api:2.14.0
command: ["./manage", "migrate"]
envFrom:
- secretRef:
name: orders-db
If migrations must stay in an init container, use a tool that takes a database lock so concurrent runs wait instead of colliding.
Permissions on volumes¶
Prefer fsGroup over a root chown init container:
securityContext:
runAsUser: 10001
runAsGroup: 10001
fsGroup: 10001
fsGroupChangePolicy: OnRootMismatch # skip recursive changes when ownership is already correct
Native sidecars are different¶
Init containers with restartPolicy: Always are sidecar containers. They start in order like init containers but keep running alongside the app instead of blocking it. If a log shipper or proxy was accidentally defined as a regular init container, the pod waits forever for it to exit — add restartPolicy: Always to make it a sidecar.
Prevention¶
- Give every init container a single responsibility and a clear failure message.
- Put timeouts on waits, and prefer application-level retries.
- Run schema migrations as Jobs tied to the release, not per pod.
- Keep init container images small and pinned, and pull them from the same trusted registry as the app.
Production Considerations¶
- Init failures block scale-ups too. During a traffic spike, new replicas stuck in
Initadd no capacity. Alert on pods inInitstates for more than a few minutes. - A dependency outage becomes a deploy outage. If every new pod waits for a downstream service, you can't roll out a fix while that service is down. Application retries avoid this coupling.
- Restarting the pod re-runs all init containers, including one-time setup steps. Make them idempotent.
Related Problems¶
- CrashLoopBackOff — the same back-off, for app containers
- ImagePullBackOff —
Init:ImagePullBackOff - Service and DNS problems
- Pods: init containers and lifecycle
- All pod startup errors