# Deployment Strategies

## What You'll Learn

- How `RollingUpdate` and `Recreate` actually behave, pod by pod
- How to tune `maxSurge` and `maxUnavailable` for your availability and cost trade-off
- How `kubectl rollout status/history/undo` fit into a real release, and where blue-green and canary sit relative to a plain Deployment
- How PodDisruptionBudgets keep a workload available during node drains and cluster upgrades

## Why This Matters

A Deployment's `strategy` field is the difference between "users never notice a release happened" and "half your requests 503 for ninety seconds." Most production incidents from deployments aren't caused by bad code — they're caused by a rollout strategy nobody tuned past the default.

## Mental Model

A Deployment doesn't update pods directly. It manages ReplicaSets, and a rollout is really: create a new ReplicaSet, scale it up, scale the old one down, according to the strategy. `RollingUpdate` (the default) does this gradually and keeps the app available throughout; `Recreate` kills every old pod before starting any new one.

| Strategy | Old pods removed | New pods created | Downtime | Use when |
| --- | --- | --- | --- | --- |
| `RollingUpdate` | Gradually, as new pods pass readiness | Gradually, up to `maxSurge` over desired count | None (if probes are correct) | Default — almost everything stateless |
| `Recreate` | All at once, before any new pod starts | Only after all old pods are gone | Yes, by design | The app can't run two versions at once (schema-incompatible singleton, exclusive port/lock) |

### RollingUpdate, tuned

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout-api
spec:
  replicas: 6
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 2          # up to 8 pods running briefly
      maxUnavailable: 0    # never drop below 6 ready pods
  minReadySeconds: 10       # pod must stay Ready this long before counting as "up"
  selector:
    matchLabels:
      app: checkout-api
  template:
    metadata:
      labels:
        app: checkout-api
    spec:
      containers:
        - name: checkout-api
          image: registry.example.com/checkout-api:1.14.2
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /healthz/ready
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 5
          resources:
            requests:
              cpu: 250m
              memory: 256Mi
            limits:
              cpu: 500m
              memory: 512Mi
```

- `maxSurge` — how many pods *above* `replicas` can exist during the rollout. Higher = faster rollout, more capacity needed.
- `maxUnavailable` — how many pods *below* `replicas` are tolerated during the rollout. `0` guarantees full capacity throughout, at the cost of needing headroom for `maxSurge` extra pods.
- Setting both to sensible non-zero values (e.g. `maxSurge: 25%`, `maxUnavailable: 25%`, the actual defaults) balances rollout speed against cluster headroom.
- A rollout only proceeds pod-by-pod as fast as the **readiness probe** says the new pods are healthy — a broken readiness probe silently stalls or fast-tracks a bad rollout.

Percentages are resolved against `replicas` with the same rules the Deployment controller uses: `maxSurge` rounds up, `maxUnavailable` rounds down. Try your own values:

**Rolling update calculator** (needs JavaScript). With 10 replicas and the default 25%/25%, `maxSurge` resolves to 3 and `maxUnavailable` to 2: at most 13 Pods exist at once, and at least 8 stay available.

### Managing a rollout

```bash
kubectl apply -f checkout-api-deployment.yaml
kubectl rollout status deployment/checkout-api
kubectl rollout history deployment/checkout-api
kubectl rollout history deployment/checkout-api --revision=3
kubectl rollout undo deployment/checkout-api
kubectl rollout undo deployment/checkout-api --to-revision=2
kubectl rollout restart deployment/checkout-api
```

`rollout status` blocks until the rollout finishes or fails — use it in CI to gate the next pipeline step. `rollout undo` works because Kubernetes keeps old ReplicaSets around (`revisionHistoryLimit`, default 10) — undo just scales the previous ReplicaSet back up and the current one down, which is itself a RollingUpdate.

```
flowchart LR
    A[Old ReplicaSet: 6/6 ready] -->|apply new image| B[New ReplicaSet scales up]
    B -->|maxSurge allows +2| C[8 pods total, new ones become Ready]
    C -->|old pods scaled down| D[Old ReplicaSet: 0/0]
    D --> E[New ReplicaSet: 6/6 ready]
    E -->|rollout undo| A
```

## PodDisruptionBudgets: Availability During Maintenance

A rollout is a disruption you start. Node drains, cluster upgrades, Cluster Autoscaler scale-downs, and spot reclaims are disruptions the **platform** starts, and a Deployment's `maxUnavailable` doesn't apply to them. A **PodDisruptionBudget (PDB)** is how a workload tells the cluster how many of its Pods may be taken down voluntarily at once:

```yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: checkout-api
spec:
  minAvailable: 5            # or: maxUnavailable: 1
  selector:
    matchLabels:
      app: checkout-api
  unhealthyPodEvictionPolicy: AlwaysAllow   # let drains evict Pods that are already broken
```

`kubectl drain`, managed node upgrades, and the autoscalers all use the **Eviction API**, which checks every matching PDB. If evicting a Pod would drop the workload below `minAvailable`, the eviction is refused and retried until another replica is ready elsewhere. With 6 replicas and `minAvailable: 5`, nodes drain one checkout Pod at a time.

```bash
kubectl get pdb -n production
# NAME           MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
# checkout-api   5               N/A               1                     12d
```

`ALLOWED DISRUPTIONS` is the number to watch. If it's `0` for long, drains and upgrades stall on this workload.

| Guideline | Why |
| --- | --- |
| Prefer `maxUnavailable: 1` for most Deployments | Keeps working as replica counts change; `minAvailable` pinned to a number can block all drains after a scale-down |
| Never set a PDB that allows zero disruptions (`minAvailable` equal to replicas, or `maxUnavailable: 0`) | Every node drain and cluster upgrade hangs on it forever |
| Single-replica workloads can't be protected by a PDB | The only way to keep them available during maintenance is to run a second replica |
| PDBs don't cover involuntary disruptions | A node crash or kernel panic ignores PDBs; replicas spread across nodes and zones are what protect against those |
| Set `unhealthyPodEvictionPolicy: AlwaysAllow` | Otherwise a crash-looping Pod counts against the budget and blocks the drain that might fix it |

## Blue-Green and Canary, Conceptually

`RollingUpdate` mixes old and new pods behind the same Service the entire time — fine for most apps, but it means both versions serve traffic simultaneously and you can't instantly cut back to 100% old version if something's wrong.

- **Blue-green** runs two full environments (blue = current, green = new) and switches traffic all at once, usually by repointing a Service selector or load balancer. Rollback is instant — flip back to blue — but you pay for two full environments during the switch.
- **Canary** sends a small percentage of traffic to the new version, watches error rates/latency, then ramps up gradually. It catches bad releases with minimal blast radius, but needs traffic-splitting infrastructure a plain Deployment doesn't have.

A Deployment alone can approximate both crudely (two Deployments + a Service selector swap for blue-green; two Deployments with proportional replica counts for a rough canary), but real weighted traffic splitting, automated analysis, and automated rollback need a progressive-delivery tool.

> [!NOTE]
>
> **Where the full walkthroughs live**
>
> Operational, tool-based walkthroughs with Argo Rollouts and Flagger are in [Progressive Delivery: Canary and Blue-Green](<https://computecentral.in/kubernetes/cicd-and-gitops/03-progressive-delivery-canary-and-blue-green/index.md>). A real-world postmortem of both patterns is in [Blue-Green and Canary Releases](<https://computecentral.in/kubernetes/case-studies/02-blue-green-and-canary-releases/index.md>).

## What Most Engineers Miss

- **The two percentages round in opposite directions.** `maxSurge` rounds up and `maxUnavailable` rounds down, so with 3 replicas the default 25%/25% means up to 1 extra Pod and **0** unavailable. Small Deployments roll one surge Pod at a time.
- **If both round to zero, the controller quietly allows one unavailable Pod.** The API rejects `maxSurge: 0` with `maxUnavailable: 0`, but `maxSurge: 0%` with `maxUnavailable: 25%` on 3 replicas rounds both to 0, and the controller then sets `maxUnavailable` to 1. A config meant to never drop capacity takes a Pod out.
- **A stuck rollout is reported, not rolled back.** After `progressDeadlineSeconds` (default 600), the Deployment's `Progressing` condition turns `False` with reason `ProgressDeadlineExceeded`, and that's all Kubernetes does: the new Pods stay. Your pipeline has to watch for it and run `kubectl rollout undo`, or use [Argo Rollouts or Flagger](<https://computecentral.in/kubernetes/cicd-and-gitops/03-progressive-delivery-canary-and-blue-green/index.md>).
- **Terminating Pods can still receive traffic.** Removing a Pod from Service endpoints and sending it SIGTERM happen at the same time, so load balancers and kube-proxy can route a few more requests to a Pod that's shutting down. A short `preStop` delay closes the gap, and since v1.34 the built-in `sleep` action does it without a `sleep` binary in the image:

  ```yaml
  lifecycle:
    preStop:
      sleep:
        seconds: 5
  ```
- **`minReadySeconds` defaults to 0.** A Pod that turns Ready and crashes two seconds later has already been counted as available, and the rollout has moved on. Set it to a few seconds longer than your app's typical crash-on-start time.
- **Undo only reaches back `revisionHistoryLimit` revisions** (default 10). Set it to 0 to save a few objects, and `kubectl rollout undo` has nothing to go back to.

## Common Mistakes

- Setting `maxUnavailable: 0` and `maxSurge: 0` at the same time — the API rejects the Deployment (`may not be 0 when maxSurge is 0`), because a rollout with no room to add or remove Pods could never progress.
- No readiness probe (or a probe that always passes) — the rollout has no real signal, so a broken new pod gets marked ready and takes production traffic immediately.
- Using `Recreate` out of habit "to be safe" — it introduces downtime a `RollingUpdate` with `maxUnavailable: 0` doesn't need.
- Confusing `kubectl rollout restart` (re-rolls current pods with the same spec, e.g. to pick up a changed Secret) with `kubectl rollout undo` (reverts to a previous revision).
- Assuming a successful `rollout status` means the release is actually healthy — it only means pods are Ready per the probe; it says nothing about real user-facing error rates.

## Check Yourself

> [!NOTE]
>
> **Defaults on a small Deployment**
>
> A Deployment has `replicas: 3` and the default strategy (`maxSurge: 25%`, `maxUnavailable: 25%`). What do the two limits resolve to?
>
> - [ ] Surge 0, unavailable 0
> - [x] Surge 1, unavailable 0
> - [ ] Surge 1, unavailable 1
>
> 25% of 3 is 0.75. `maxSurge` rounds up to 1, `maxUnavailable` rounds down to 0, so the rollout adds one new Pod, waits for it to be Ready, then removes an old one.

> [!NOTE]
>
> **Past the deadline**
>
> A new image crashes on start, and the rollout exceeds `progressDeadlineSeconds`. What does Kubernetes do?
>
> - [ ] Rolls back to the previous ReplicaSet automatically
> - [x] Marks the Deployment `Progressing=False` (`ProgressDeadlineExceeded`) and leaves everything as it is
> - [ ] Deletes the new ReplicaSet
>
> Kubernetes only reports a stalled rollout. Rolling back is up to you, your pipeline, or a progressive delivery controller.

> [!NOTE]
>
> **Rounded to zero**
>
> `replicas: 4`, `maxSurge: 0`, `maxUnavailable: 10%`. How many Pods can be unavailable during the rollout?
>
> - [ ] 0, so the rollout hangs
> - [x] 1: 10% of 4 rounds down to 0, and with both limits at 0 the controller uses 1
> - [ ] The API rejects the Deployment
>
> The API only rejects literal zeros for both. When rounding produces zero for both, the controller allows one unavailable Pod so the rollout can move.

## Interview Questions

- Walk through exactly what happens, pod by pod, during a `RollingUpdate` with `maxSurge: 1, maxUnavailable: 0`.
- A cluster upgrade is stuck draining one node. How would a PodDisruptionBudget cause that, and how do you find which one?
- Why would you choose `Recreate` over `RollingUpdate`, and what's the cost?
- How does `kubectl rollout undo` work under the hood, given Deployments don't store "old pods" directly?
- How does a canary release differ from what a Deployment's `RollingUpdate` already gives you?

See [Interview Prep](<https://computecentral.in/kubernetes/interview-prep/index.md>) for full answers.

## Related

- [Autoscaling: HPA, VPA, and Cluster Autoscaler](<https://computecentral.in/kubernetes/workloads-and-scheduling/06-autoscaling/index.md>)

## Next

Continue to [StatefulSets](<https://computecentral.in/kubernetes/workloads-and-scheduling/02-statefulsets/index.md>) for workloads that need stable identity instead of interchangeable pods.
