Deployment Strategies¶
What You'll Learn¶
- How
RollingUpdateandRecreateactually behave, pod by pod - How to tune
maxSurgeandmaxUnavailablefor your availability and cost trade-off - How
kubectl rollout status/history/undofit into a real release, and where blue-green and canary sit relative to a plain Deployment - How PodDisruptionBudgets keep a workload available during node drains and cluster upgrades
Why This Matters¶
A Deployment's strategy field is the difference between "users never notice a release happened" and "half your requests 503 for ninety seconds." Most production incidents from deployments aren't caused by bad code — they're caused by a rollout strategy nobody tuned past the default.
Mental Model¶
A Deployment doesn't update pods directly. It manages ReplicaSets, and a rollout is really: create a new ReplicaSet, scale it up, scale the old one down, according to the strategy. RollingUpdate (the default) does this gradually and keeps the app available throughout; Recreate kills every old pod before starting any new one.
| Strategy | Old pods removed | New pods created | Downtime | Use when |
|---|---|---|---|---|
RollingUpdate |
Gradually, as new pods pass readiness | Gradually, up to maxSurge over desired count |
None (if probes are correct) | Default — almost everything stateless |
Recreate |
All at once, before any new pod starts | Only after all old pods are gone | Yes, by design | The app can't run two versions at once (schema-incompatible singleton, exclusive port/lock) |
RollingUpdate, tuned¶
apiVersion: apps/v1
kind: Deployment
metadata:
name: checkout-api
spec:
replicas: 6
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 2 # up to 8 pods running briefly
maxUnavailable: 0 # never drop below 6 ready pods
minReadySeconds: 10 # pod must stay Ready this long before counting as "up"
selector:
matchLabels:
app: checkout-api
template:
metadata:
labels:
app: checkout-api
spec:
containers:
- name: checkout-api
image: registry.example.com/checkout-api:1.14.2
ports:
- containerPort: 8080
readinessProbe:
httpGet:
path: /healthz/ready
port: 8080
initialDelaySeconds: 5
periodSeconds: 5
resources:
requests:
cpu: 250m
memory: 256Mi
limits:
cpu: 500m
memory: 512Mi
maxSurge— how many pods abovereplicascan exist during the rollout. Higher = faster rollout, more capacity needed.maxUnavailable— how many pods belowreplicasare tolerated during the rollout.0guarantees full capacity throughout, at the cost of needing headroom formaxSurgeextra pods.- Setting both to sensible non-zero values (e.g.
maxSurge: 25%,maxUnavailable: 25%, the actual defaults) balances rollout speed against cluster headroom. - A rollout only proceeds pod-by-pod as fast as the readiness probe says the new pods are healthy — a broken readiness probe silently stalls or fast-tracks a bad rollout.
Percentages are resolved against replicas with the same rules the Deployment controller uses: maxSurge rounds up, maxUnavailable rounds down. Try your own values:
Rolling update calculator (needs JavaScript). With 10 replicas and the default 25%/25%, maxSurge resolves to 3 and maxUnavailable to 2: at most 13 Pods exist at once, and at least 8 stay available.
Managing a rollout¶
kubectl apply -f checkout-api-deployment.yaml
kubectl rollout status deployment/checkout-api
kubectl rollout history deployment/checkout-api
kubectl rollout history deployment/checkout-api --revision=3
kubectl rollout undo deployment/checkout-api
kubectl rollout undo deployment/checkout-api --to-revision=2
kubectl rollout restart deployment/checkout-api
rollout status blocks until the rollout finishes or fails — use it in CI to gate the next pipeline step. rollout undo works because Kubernetes keeps old ReplicaSets around (revisionHistoryLimit, default 10) — undo just scales the previous ReplicaSet back up and the current one down, which is itself a RollingUpdate.
flowchart LR
A[Old ReplicaSet: 6/6 ready] -->|apply new image| B[New ReplicaSet scales up]
B -->|maxSurge allows +2| C[8 pods total, new ones become Ready]
C -->|old pods scaled down| D[Old ReplicaSet: 0/0]
D --> E[New ReplicaSet: 6/6 ready]
E -->|rollout undo| A
PodDisruptionBudgets: Availability During Maintenance¶
A rollout is a disruption you start. Node drains, cluster upgrades, Cluster Autoscaler scale-downs, and spot reclaims are disruptions the platform starts, and a Deployment's maxUnavailable doesn't apply to them. A PodDisruptionBudget (PDB) is how a workload tells the cluster how many of its Pods may be taken down voluntarily at once:
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: checkout-api
spec:
minAvailable: 5 # or: maxUnavailable: 1
selector:
matchLabels:
app: checkout-api
unhealthyPodEvictionPolicy: AlwaysAllow # let drains evict Pods that are already broken
kubectl drain, managed node upgrades, and the autoscalers all use the Eviction API, which checks every matching PDB. If evicting a Pod would drop the workload below minAvailable, the eviction is refused and retried until another replica is ready elsewhere. With 6 replicas and minAvailable: 5, nodes drain one checkout Pod at a time.
kubectl get pdb -n production
# NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE
# checkout-api 5 N/A 1 12d
ALLOWED DISRUPTIONS is the number to watch. If it's 0 for long, drains and upgrades stall on this workload.
| Guideline | Why |
|---|---|
Prefer maxUnavailable: 1 for most Deployments |
Keeps working as replica counts change; minAvailable pinned to a number can block all drains after a scale-down |
Never set a PDB that allows zero disruptions (minAvailable equal to replicas, or maxUnavailable: 0) |
Every node drain and cluster upgrade hangs on it forever |
| Single-replica workloads can't be protected by a PDB | The only way to keep them available during maintenance is to run a second replica |
| PDBs don't cover involuntary disruptions | A node crash or kernel panic ignores PDBs; replicas spread across nodes and zones are what protect against those |
Set unhealthyPodEvictionPolicy: AlwaysAllow |
Otherwise a crash-looping Pod counts against the budget and blocks the drain that might fix it |
Blue-Green and Canary, Conceptually¶
RollingUpdate mixes old and new pods behind the same Service the entire time — fine for most apps, but it means both versions serve traffic simultaneously and you can't instantly cut back to 100% old version if something's wrong.
- Blue-green runs two full environments (blue = current, green = new) and switches traffic all at once, usually by repointing a Service selector or load balancer. Rollback is instant — flip back to blue — but you pay for two full environments during the switch.
- Canary sends a small percentage of traffic to the new version, watches error rates/latency, then ramps up gradually. It catches bad releases with minimal blast radius, but needs traffic-splitting infrastructure a plain Deployment doesn't have.
A Deployment alone can approximate both crudely (two Deployments + a Service selector swap for blue-green; two Deployments with proportional replica counts for a rough canary), but real weighted traffic splitting, automated analysis, and automated rollback need a progressive-delivery tool.
Where the full walkthroughs live
Operational, tool-based walkthroughs with Argo Rollouts and Flagger are in Progressive Delivery: Canary and Blue-Green. A real-world postmortem of both patterns is in Blue-Green and Canary Releases.
What Most Engineers Miss¶
- The two percentages round in opposite directions.
maxSurgerounds up andmaxUnavailablerounds down, so with 3 replicas the default 25%/25% means up to 1 extra Pod and 0 unavailable. Small Deployments roll one surge Pod at a time. - If both round to zero, the controller quietly allows one unavailable Pod. The API rejects
maxSurge: 0withmaxUnavailable: 0, butmaxSurge: 0%withmaxUnavailable: 25%on 3 replicas rounds both to 0, and the controller then setsmaxUnavailableto 1. A config meant to never drop capacity takes a Pod out. - A stuck rollout is reported, not rolled back. After
progressDeadlineSeconds(default 600), the Deployment'sProgressingcondition turnsFalsewith reasonProgressDeadlineExceeded, and that's all Kubernetes does: the new Pods stay. Your pipeline has to watch for it and runkubectl rollout undo, or use Argo Rollouts or Flagger. -
Terminating Pods can still receive traffic. Removing a Pod from Service endpoints and sending it SIGTERM happen at the same time, so load balancers and kube-proxy can route a few more requests to a Pod that's shutting down. A short
preStopdelay closes the gap, and since v1.34 the built-insleepaction does it without asleepbinary in the image: -
minReadySecondsdefaults to 0. A Pod that turns Ready and crashes two seconds later has already been counted as available, and the rollout has moved on. Set it to a few seconds longer than your app's typical crash-on-start time. - Undo only reaches back
revisionHistoryLimitrevisions (default 10). Set it to 0 to save a few objects, andkubectl rollout undohas nothing to go back to.
Common Mistakes¶
- Setting
maxUnavailable: 0andmaxSurge: 0at the same time — the API rejects the Deployment (may not be 0 when maxSurge is 0), because a rollout with no room to add or remove Pods could never progress. - No readiness probe (or a probe that always passes) — the rollout has no real signal, so a broken new pod gets marked ready and takes production traffic immediately.
- Using
Recreateout of habit "to be safe" — it introduces downtime aRollingUpdatewithmaxUnavailable: 0doesn't need. - Confusing
kubectl rollout restart(re-rolls current pods with the same spec, e.g. to pick up a changed Secret) withkubectl rollout undo(reverts to a previous revision). - Assuming a successful
rollout statusmeans the release is actually healthy — it only means pods are Ready per the probe; it says nothing about real user-facing error rates.
Check Yourself¶
Defaults on a small Deployment
A Deployment has replicas: 3 and the default strategy (maxSurge: 25%, maxUnavailable: 25%). What do the two limits resolve to?
- Surge 0, unavailable 0
- Surge 1, unavailable 0
- Surge 1, unavailable 1
25% of 3 is 0.75. maxSurge rounds up to 1, maxUnavailable rounds down to 0, so the rollout adds one new Pod, waits for it to be Ready, then removes an old one.
Past the deadline
A new image crashes on start, and the rollout exceeds progressDeadlineSeconds. What does Kubernetes do?
- Rolls back to the previous ReplicaSet automatically
- Marks the Deployment
Progressing=False(ProgressDeadlineExceeded) and leaves everything as it is - Deletes the new ReplicaSet
Kubernetes only reports a stalled rollout. Rolling back is up to you, your pipeline, or a progressive delivery controller.
Rounded to zero
replicas: 4, maxSurge: 0, maxUnavailable: 10%. How many Pods can be unavailable during the rollout?
- 0, so the rollout hangs
- 1: 10% of 4 rounds down to 0, and with both limits at 0 the controller uses 1
- The API rejects the Deployment
The API only rejects literal zeros for both. When rounding produces zero for both, the controller allows one unavailable Pod so the rollout can move.
Interview Questions¶
- Walk through exactly what happens, pod by pod, during a
RollingUpdatewithmaxSurge: 1, maxUnavailable: 0. - A cluster upgrade is stuck draining one node. How would a PodDisruptionBudget cause that, and how do you find which one?
- Why would you choose
RecreateoverRollingUpdate, and what's the cost? - How does
kubectl rollout undowork under the hood, given Deployments don't store "old pods" directly? - How does a canary release differ from what a Deployment's
RollingUpdatealready gives you?
See Interview Prep for full answers.
Related¶
Next¶
Continue to StatefulSets for workloads that need stable identity instead of interchangeable pods.