Cluster Upgrades¶
What You'll Learn¶
- The version skew policy that governs how far apart your control plane and kubelets are allowed to drift
- The correct
kubeadm upgradeorder: control plane first, then nodes one at a time - How EKS, GKE, and AKS change — and simplify — this process on managed control planes
Why This Matters¶
Kubernetes ships a new minor version roughly every four months, and each minor version is supported for about 14 months. A cluster that never upgrades eventually falls off supported versions entirely, loses access to security patches, and accumulates enough version drift that the eventual forced upgrade becomes far riskier than a series of small ones would have been.
Mental Model¶
Kubernetes deliberately allows some version drift between components, but only within specific bounds — and the bounds all point in the same direction: the control plane must never be older than the components below it.
The version skew policy¶
| Component pair | Maximum skew allowed |
|---|---|
kube-apiserver instances (HA cluster) |
Must all be within 1 minor version of each other |
kubelet vs. kube-apiserver |
kubelet may be up to 3 minor versions older (as of recent policy; historically 2) — but never newer |
kube-controller-manager/kube-scheduler vs. kube-apiserver |
Must not be newer than the API server; may be up to 1 minor version older |
kubectl vs. kube-apiserver |
May be 1 minor version older or newer |
The rule that matters most in practice: kubelets can lag behind the API server, but never lead it. This is exactly why upgrades always go control plane first — upgrading a kubelet to a newer minor version than the API server it talks to is unsupported and can break in unpredictable ways.
flowchart LR
A["v1.36 API server"] -->|supports kubelets down to| B["v1.35, v1.34, v1.33"]
A -.->|never supports| C["v1.37 kubelet"]
The kubeadm upgrade workflow¶
Upgrades happen one minor version at a time — you cannot skip from 1.35 straight to 1.37, you go 1.35 → 1.36 → 1.37, verifying health at each step.
0. Point the package repository at the new minor version (every node)
The pkgs.k8s.io repositories are per minor version, so apt can't see 1.37 packages until you change the repository line. Skipping this is the most common reason an upgrade "can't find" the new version:
sudo sed -i 's#/v1.36/#/v1.37/#' /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
apt-cache madison kubeadm | head -3 # confirm 1.37.x packages are now visible
Before touching anything: take an etcd snapshot (Backup and Restore), and check that nothing still uses an API removed in the target version (kubent, pluto, or the API server's apiserver_requested_deprecated_apis metric will tell you).
1. Upgrade the first control-plane node
# On the first control-plane node
sudo apt-mark unhold kubeadm
sudo apt-get install -y kubeadm=1.37.0-1.1
sudo apt-mark hold kubeadm
sudo kubeadm upgrade plan # shows what will change, sanity-checks skew
sudo kubeadm upgrade apply v1.37.0
sudo apt-mark unhold kubelet kubectl
sudo apt-get install -y kubelet=1.37.0-1.1 kubectl=1.37.0-1.1
sudo apt-mark hold kubelet kubectl
sudo systemctl daemon-reload
sudo systemctl restart kubelet
2. Upgrade remaining control-plane nodes
# On each additional control-plane node
sudo kubeadm upgrade node
# then upgrade kubelet/kubectl packages exactly as above
3. Upgrade worker nodes, one at a time
# From a machine with kubectl access
kubectl drain node-worker-1 --ignore-daemonsets --delete-emptydir-data
# On node-worker-1
sudo kubeadm upgrade node
sudo apt-get install -y kubelet=1.37.0-1.1 kubectl=1.37.0-1.1
sudo systemctl daemon-reload
sudo systemctl restart kubelet
# Back on the kubectl machine
kubectl uncordon node-worker-1
kubectl get nodes # confirm Ready at the new version before moving to the next node
Repeat the drain → upgrade → uncordon cycle for every worker node, one at a time, confirming health before moving on. This is the same cordon/drain discipline from Node Management applied to an entire fleet.
Never upgrade all nodes simultaneously
Draining and upgrading nodes one at a time is what keeps the application available during the upgrade. Upgrading every node at once is functionally a full-cluster outage.
Before You Upgrade: Checks That Catch Real Outages¶
Kubernetes doesn't support downgrades. If an upgrade breaks something, the way back is a restore from backup, so the checks happen before, not after:
- Read the Urgent Upgrade Notes at the top of the target version's changelog. They list what the release removes or changes in a way that can break you.
-
Search workloads and add-on configuration for labels, taints, and fields the release removes. Custom resources count: Reddit's outage hid in Calico's route reflector config, not in any Deployment.
-
Ask the API server whether anything still calls deprecated APIs. The counter covers every client since the API server last started, including ones not in your Git repository:
-
Confirm every node runs cgroup v2 (
stat -fc %T /sys/fs/cgroupprintscgroup2fs). Since v1.35 the kubelet refuses to start on cgroup v1 unlessfailCgroupV1: falseis set. - Check that the CNI, CSI drivers, ingress or Gateway controller, and service mesh support the target version.
- Look for PodDisruptionBudgets that will block drains: any PDB showing
ALLOWED DISRUPTIONS 0inkubectl get pdb -Awill stall a node. - Take an etcd snapshot and know how to restore it (see Backup and Restore).
- Rehearse on a cluster with the same add-ons. A clean cluster built last week is not the one that will surprise you.
Real incident: Reddit, March 2023
Reddit's upgrade of its oldest, largest cluster from 1.23 to 1.24 removed the node-role.kubernetes.io/master label. Calico's route reflectors were selected by that label, so they vanished, Pod networking collapsed, and Reddit was down for 314 minutes before the team restored from backup. The full story.
As of October 2026 the upstream project supports 1.37, 1.36, and 1.35, and 1.34 reaches end of life on October 27, 2026. A cluster on 1.34 or older is out of upstream patch support unless a vendor extends it.
How Managed Platforms Change This¶
| Platform | Control plane upgrade | Node upgrade |
|---|---|---|
Self-managed (kubeadm) |
Fully manual, as above | Fully manual, one node at a time |
| Amazon EKS | One eksctl upgrade cluster / console click; AWS handles etcd and API server |
Managed node groups support rolling upgrade via eksctl upgrade nodegroup; self-managed node groups still need manual draining |
| Google GKE | Automatic by default (configurable maintenance windows); can be manual | Node auto-upgrade is on by default, respecting PodDisruptionBudgets during rolling replacement |
| Azure AKS | az aks upgrade triggers a managed, orchestrated control-plane upgrade |
Node pools upgrade via az aks nodepool upgrade, with configurable surge and max-unavailable settings |
The version skew policy still applies underneath all three managed platforms — they just automate the sequencing and safety checks that kubeadm upgrade makes you run by hand. The trade-off is control: managed platforms restrict how far you can defer an upgrade, and deferring can cost money:
- EKS gives each version about 14 months of standard support, then extended support, which clusters enter automatically unless you opt out. It's billed at $0.60 per cluster per hour instead of $0.10, six times the standard rate, so clusters left on old versions quietly become one of the larger line items on the bill.
- GKE enrolls clusters in release channels (Rapid, Regular, Stable, Extended) and upgrades them automatically within your maintenance windows and exclusions.
- AKS offers a paid long-term support option for selected versions.
Budget for a routine upgrade every quarter or so; it's far less work than a forced multi-version jump.
Common Mistakes¶
- Upgrading a kubelet to a newer minor version than the control plane — unsupported, and can cause API compatibility failures.
- Skipping minor versions (1.35 straight to 1.37) instead of stepping through each one.
- Upgrading all worker nodes at once instead of one at a time with drain/uncordon between each.
- Forgetting to
apt-mark hold/pin the package versions, letting an unrelatedapt upgradesilently bump Kubernetes components out of band. - Assuming a managed platform's "automatic" node upgrade doesn't need
PodDisruptionBudgets — it still respects them, and a misconfigured PDB can still stall a managed upgrade.
Interview Questions¶
- What is the Kubernetes version skew policy, and which direction is the drift allowed to go?
- Why must the control plane always be upgraded before the kubelets?
- Walk through the full
kubeadm upgradesequence for a 3-control-plane, 5-worker cluster. - How does upgrading an EKS-managed node group differ operationally from upgrading self-managed
kubeadmworker nodes?
See Interview Prep for full answers.
Next¶
Continue to Backup and Restore — take a fresh etcd snapshot before every upgrade, so a bad one is always recoverable.