# Events and Debugging

To debug a Kubernetes pod, work from the outside in: `kubectl get pods` for its status, `kubectl describe pod` for its events and exit codes, `kubectl logs --previous` for what the app printed before it died, and `kubectl debug` for a shell when the image has none. The status column usually tells you where to start:

| What you see | Run first | Usual cause | Full guide |
| --- | --- | --- | --- |
| `Pending` | `kubectl describe pod <pod>` (Events) | No node has enough CPU or memory, a taint or affinity rule blocks it, or a PVC isn't bound | [Pod stuck in Pending](<https://computecentral.in/kubernetes/troubleshooting/pod-pending/index.md>) |
| `ImagePullBackOff` / `ErrImagePull` | `kubectl describe pod <pod>` | Wrong image name or tag, private registry without `imagePullSecrets` | [ImagePullBackOff](<https://computecentral.in/kubernetes/troubleshooting/imagepullbackoff/index.md>) |
| `CrashLoopBackOff` | `kubectl logs <pod> --previous` | The app exits on startup: bad config, missing secret, failing liveness probe | [CrashLoopBackOff](<https://computecentral.in/kubernetes/troubleshooting/crashloopbackoff/index.md>) |
| `OOMKilled`, exit code 137 | `kubectl describe pod <pod>` (Last State) | Memory limit lower than the app needs | [OOMKilled](<https://computecentral.in/kubernetes/troubleshooting/oomkilled/index.md>) |
| `Init:CrashLoopBackOff` | `kubectl logs <pod> -c <init-container>` | An init container fails, often waiting for a dependency | [Init container failures](<https://computecentral.in/kubernetes/troubleshooting/init-container-failures/index.md>) |
| `Running` but not serving | `kubectl get endpointslices -l kubernetes.io/service-name=<svc>` | Readiness probe failing, or Service selector doesn't match the pod labels | [Service and DNS problems](<https://computecentral.in/kubernetes/troubleshooting/02-networking-and-service-problems/index.md>) |
| Node `NotReady` | `kubectl describe node <node>` | kubelet down, disk or memory pressure, CNI not running | [Node and control plane problems](<https://computecentral.in/kubernetes/troubleshooting/04-cluster-and-node-problems/index.md>) |

## What You'll Learn

- The five core `kubectl` debugging tools, and what each one actually reveals that the others don't
- Ephemeral debug containers — how to get a shell into a pod that has no shell of its own
- A repeatable, ordered debugging methodology you can apply to almost any broken workload

## Why This Matters

Debugging Kubernetes under pressure goes much faster with a fixed sequence than with random `kubectl` commands fired in whatever order comes to mind. The tools in this page are the same handful you'll use for nearly every incident — the skill is knowing which one answers which question, and in what order to reach for them so you're not guessing.

## Mental Model

> Diagnose from the outside in: what does Kubernetes *think* the pod's state is, then what *happened* to get it there, then what is the *application itself* saying, and only then go *inside* the container if you still don't have an answer.

```
flowchart LR
    A[kubectl get pods] --> B[kubectl describe pod]
    B --> C[kubectl get events]
    C --> D[kubectl logs]
    D --> E[kubectl exec / debug]
```

## How It Works

### Step 1 — Pod status

```bash
kubectl get pods -n production -o wide
```

Start here: is it `Pending` (not scheduled yet), `CrashLoopBackOff` (starting and dying repeatedly), `ImagePullBackOff` (can't pull the image), or `Running` but just not behaving correctly? The status alone usually narrows which of the next steps matters most.

### Step 2 — `kubectl describe`

```bash
kubectl describe pod my-pod -n production
```

`describe` surfaces what plain `get` doesn't: the scheduling decision (or why it failed), container state transitions with exit codes and reasons, resource requests/limits versus what the node actually has available, mounted volumes, and — critically — an embedded feed of recent Events scoped to that object.

### Step 3 — `kubectl get events`

```bash
# Cluster-wide, in chronological order
kubectl get events -n production --sort-by=.metadata.creationTimestamp

# Live stream
kubectl get events -n production --watch

# Scoped to one object
kubectl get events -n production --field-selector involvedObject.name=my-pod
```

Events capture things `describe` on a single pod won't show in full context — a FailedScheduling event across all pending pods, a node's MemoryPressure condition, a volume attach failure. Events are also **short-lived** (retained roughly an hour by default) — if you're investigating something that happened earlier, the Events API may have already garbage collected the evidence, which is exactly why cluster-wide event/log aggregation matters for anything beyond immediate triage.

### Step 4 — Logs

Covered in full in [Logging](<https://computecentral.in/kubernetes/observability/02-logging/index.md>) — `kubectl logs`, `--previous` for a crashed container's last output, `-c` for a specific container in a multi-container pod.

### Step 5 — Getting inside: exec, ephemeral debug containers, port-forward, cp

```bash
# Shell into a running container that has one
kubectl exec -it my-pod -- /bin/sh

# A specific container in a multi-container pod
kubectl exec -it my-pod -c sidecar -- /bin/sh
```

Many production images (distroless, `scratch`-based) deliberately ship **without** a shell — `kubectl exec` into them fails outright. **Ephemeral debug containers**, via `kubectl debug`, solve this by attaching a temporary container with full debugging tools directly into the target pod's process/network namespace, without modifying the pod's actual spec or restarting it:

```bash
# Attach a debug container with a full toolset to an already-running pod
kubectl debug -it my-pod --image=busybox:1.36 --target=app

# Debug a node itself by launching a privileged pod in its namespace
kubectl debug node/my-node -it --image=busybox:1.36
```

Two more forms solve problems that come up constantly in real incidents:

```bash
# The pod crashes too fast to attach to: debug a COPY with a shell and a new entrypoint
kubectl debug my-pod -it --copy-to=my-pod-debug --container=app -- sh

# Need tcpdump/iptables-level access: use a debugging profile that adds NET_ADMIN/NET_RAW
kubectl debug -it my-pod --image=nicolaka/netshoot --target=app --profile=netadmin
```

`--copy-to` creates a new Pod from the original's spec (you can also swap its image with `--set-image`), so the original keeps crash-looping untouched while you poke at the copy. Delete the copy afterwards. Profiles (`general`, `baseline`, `restricted`, `netadmin`, `sysadmin`) set the debug container's security context; in a namespace enforcing the Restricted Pod Security Standard, only `restricted` is admitted. `kubectl debug node/...` mounts the node's root filesystem at `/host`, so `chroot /host` gives you the node's own tools.

`--target=app` is what makes this genuinely useful for distroless containers: the ephemeral container shares the target container's process namespace, so tools like `ps`, `netstat`, or `curl` inside the debug container can inspect the actual running application process even though the application's own image has none of those tools.

```bash
# Test connectivity to a Service from inside the cluster
kubectl port-forward service/my-service 8080:80
# then curl localhost:8080 from your machine

# Copy a file out of (or into) a container for offline inspection
kubectl cp production/my-pod:/var/log/app/error.log ./error.log
```

`port-forward` is especially useful for reaching a Service or Pod that has no external exposure at all — bypassing Ingress and LoadBalancer entirely to test straight from your local machine. `cp` is the fastest way to pull a large log file or heap dump off a container for analysis somewhere with better tooling than a shell inside the pod.

### Putting the methodology together

1. **Status** — `kubectl get pods` — what phase is it in?
2. **Describe** — `kubectl describe pod` — what does Kubernetes' own state say happened?
3. **Events** — `kubectl get events --sort-by=...` — what happened, cluster-wide, around that time?
4. **Logs** — `kubectl logs` (with `--previous` if it crashed) — what did the application itself report?
5. **Exec / debug** — `kubectl exec` or `kubectl debug` — if the above didn't explain it, get inside and look directly.

This exact sequence is what the [Troubleshooting](<https://computecentral.in/kubernetes/troubleshooting/index.md>) section applies to specific symptoms — `CrashLoopBackOff`, `ImagePullBackOff`, a Service with no working endpoints — so it's worth internalizing here rather than re-deriving it under pressure each time.

## What Most Engineers Miss

- **The one-hour limit is a flag.** The API server keeps Events for `--event-ttl`, which defaults to `1h`. Managed control planes rarely let you change it, so if you want Events in a postmortem, run an event exporter that ships them to your log store.
- **`kubectl events` beats `kubectl get events`.** `kubectl events --for pod/my-pod --types=Warning` shows one object's warnings in time order, with no field selectors to remember.
- **One Event line can stand for hundreds.** Repeats are folded into a single Event with a count, so `Back-off restarting failed container` seen once in `describe` may have happened 300 times. Check the count before deciding something is rare.
- **Ephemeral containers can't be removed.** Each `kubectl debug` adds one to the Pod for good; they stay until the Pod is deleted. On long-lived Pods, prefer `--copy-to` once you've seen what you need.
- **Every one of these commands needs a healthy API server.** `logs`, `exec`, `debug`, and `events` all go through it. When the control plane is down, as in [OpenAI's December 2024 outage](<https://computecentral.in/kubernetes/case-studies/07-real-world-outages-and-lessons/index.md#openai-2024-telemetry-on-every-node-overloaded-the-api-servers>), you need node-level access as a break-glass path: SSH or a cloud session manager, then `crictl ps` and `crictl logs <container-id>` on the node.

## Common Mistakes

- Jumping straight to `kubectl exec` before checking `describe` and `events`, which usually already explain the problem without needing to get inside the container at all.
- Trying `kubectl exec` on a distroless or `scratch`-based image and concluding the pod is "unreachable" instead of using `kubectl debug --target` to attach an ephemeral debug container.
- Not checking `kubectl get events` quickly enough — events are short-lived, and the evidence for an intermittent issue can be gone by the time you look.
- Forgetting `--previous` on `kubectl logs` after a crash and concluding there's nothing useful in the logs.
- Modifying a pod's spec (adding a debug sidecar, changing the image) just to get a shell, when an ephemeral debug container achieves the same result with zero change to the running workload.

## Check Yourself

> [!NOTE]
>
> **The missing evidence**
>
> A Pod was evicted at 02:00. At 09:00 you run `kubectl get events` and find nothing about it. Why?
>
> - [ ] Evictions don't create Events
> - [x] Events are kept for `--event-ttl`, one hour by default, and these have expired
> - [ ] `kubectl get events` only shows the current namespace's last 100 Events
>
> Short retention is by design: Events live in etcd. Export them if you need them later.

> [!NOTE]
>
> **No shell in the image**
>
> A distroless container is misbehaving and `kubectl exec -it my-pod -- sh` fails. What gives you tools inside its process namespace without restarting it?
>
> - [ ] `kubectl exec` with `--privileged`
> - [x] `kubectl debug -it my-pod --image=busybox:1.36 --target=app`
> - [ ] `kubectl cp` a shell into the container
>
> An ephemeral container with `--target` shares the app container's process namespace, so `ps` and `/proc` show the real process.

> [!NOTE]
>
> **The control plane is down**
>
> The API servers are overloaded and every `kubectl` command times out. How do you see a container's logs?
>
> - [ ] `kubectl logs --request-timeout=5m`
> - [x] Log in to the node and use `crictl logs`, or read `/var/log/pods/` directly
> - [ ] You can't until the API server recovers
>
> The kubelet and container runtime keep running without the API server. Node access is the break-glass path, so make sure it exists before you need it.

## Interview Questions

- Walk through your debugging methodology for a pod stuck in `CrashLoopBackOff`, in order.
- How do ephemeral debug containers let you debug a distroless image that has no shell?
- Why might `kubectl get events` not show you something that happened 90 minutes ago?
- What's the difference between what `kubectl describe pod` tells you and what `kubectl logs` tells you?

See [Interview Prep](<https://computecentral.in/kubernetes/interview-prep/index.md>) for full answers.

## Next

Continue to [Cluster Administration](<https://computecentral.in/kubernetes/cluster-administration/index.md>) to see how these same signals — status, events, logs, metrics — feed into operating the cluster as a whole.
