Cluster Upgrades¶

By Sameer Alam · Updated 26 September 2026

What You'll Learn¶

• The version skew policy that governs how far apart your control plane and kubelets are allowed to drift
• The correct kubeadm upgrade order: control plane first, then nodes one at a time
• How EKS, GKE, and AKS change — and simplify — this process on managed control planes

Why This Matters¶

Kubernetes ships a new minor version roughly every four months, and each minor version is supported for about 14 months. A cluster that never upgrades eventually falls off supported versions entirely, loses access to security patches, and accumulates enough version drift that the eventual forced upgrade becomes far riskier than a series of small ones would have been.

Mental Model¶

Kubernetes deliberately allows some version drift between components, but only within specific bounds — and the bounds all point in the same direction: the control plane must never be older than the components below it.

The version skew policy¶

The rule that matters most in practice: kubelets can lag behind the API server, but never lead it. This is exactly why upgrades always go control plane first — upgrading a kubelet to a newer minor version than the API server it talks to is unsupported and can break in unpredictable ways.

flowchart LR
    A["v1.36 API server"] -->|supports kubelets down to| B["v1.35, v1.34, v1.33"]
    A -.->|never supports| C["v1.37 kubelet"]

The kubeadm upgrade workflow¶

Upgrades happen one minor version at a time — you cannot skip from 1.35 straight to 1.37, you go 1.35 → 1.36 → 1.37, verifying health at each step.

0. Point the package repository at the new minor version (every node)

The pkgs.k8s.io repositories are per minor version, so apt can't see 1.37 packages until you change the repository line. Skipping this is the most common reason an upgrade "can't find" the new version:

sudo sed -i 's#/v1.36/#/v1.37/#' /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
apt-cache madison kubeadm | head -3        # confirm 1.37.x packages are now visible

Before touching anything: take an etcd snapshot (Backup and Restore), and check that nothing still uses an API removed in the target version (kubent, pluto, or the API server's apiserver_requested_deprecated_apis metric will tell you).

1. Upgrade the first control-plane node

# On the first control-plane node
sudo apt-mark unhold kubeadm
sudo apt-get install -y kubeadm=1.37.0-1.1
sudo apt-mark hold kubeadm

sudo kubeadm upgrade plan          # shows what will change, sanity-checks skew
sudo kubeadm upgrade apply v1.37.0

sudo apt-mark unhold kubelet kubectl
sudo apt-get install -y kubelet=1.37.0-1.1 kubectl=1.37.0-1.1
sudo apt-mark hold kubelet kubectl
sudo systemctl daemon-reload
sudo systemctl restart kubelet

2. Upgrade remaining control-plane nodes

# On each additional control-plane node
sudo kubeadm upgrade node
# then upgrade kubelet/kubectl packages exactly as above

3. Upgrade worker nodes, one at a time

# From a machine with kubectl access
kubectl drain node-worker-1 --ignore-daemonsets --delete-emptydir-data

# On node-worker-1
sudo kubeadm upgrade node
sudo apt-get install -y kubelet=1.37.0-1.1 kubectl=1.37.0-1.1
sudo systemctl daemon-reload
sudo systemctl restart kubelet

# Back on the kubectl machine
kubectl uncordon node-worker-1
kubectl get nodes   # confirm Ready at the new version before moving to the next node

Repeat the drain → upgrade → uncordon cycle for every worker node, one at a time, confirming health before moving on. This is the same cordon/drain discipline from Node Management applied to an entire fleet.

How Managed Platforms Change This¶

The version skew policy still applies underneath all three managed platforms — they just automate the sequencing and safety checks that kubeadm upgrade makes you run by hand. The trade-off is control: managed platforms restrict how far you can defer an upgrade, and deferring can cost money:

• EKS gives each version about 14 months of standard support, then an optional extended support period billed at a much higher per-cluster hourly rate. Clusters left on old versions quietly become one of the larger line items on the bill.
• GKE enrolls clusters in release channels (Rapid, Regular, Stable, Extended) and upgrades them automatically within your maintenance windows and exclusions.
• AKS offers a paid long-term support option for selected versions.

Budget for a routine upgrade every quarter or so; it's far less work than a forced multi-version jump.

Common Mistakes¶

• Upgrading a kubelet to a newer minor version than the control plane — unsupported, and can cause API compatibility failures.
• Skipping minor versions (1.35 straight to 1.37) instead of stepping through each one.
• Upgrading all worker nodes at once instead of one at a time with drain/uncordon between each.
• Forgetting to apt-mark hold/pin the package versions, letting an unrelated apt upgrade silently bump Kubernetes components out of band.
• Assuming a managed platform's "automatic" node upgrade doesn't need PodDisruptionBudgets — it still respects them, and a misconfigured PDB can still stall a managed upgrade.

Interview Questions¶

• What is the Kubernetes version skew policy, and which direction is the drift allowed to go?
• Why must the control plane always be upgraded before the kubelets?
• Walk through the full kubeadm upgrade sequence for a 3-control-plane, 5-worker cluster.
• How does upgrading an EKS-managed node group differ operationally from upgrading self-managed kubeadm worker nodes?

See Interview Prep for full answers.

Next¶

Continue to Backup and Restore — take a fresh etcd snapshot before every upgrade, so a bad one is always recoverable.