Skip to content

Cluster Upgrades

What You'll Learn

  • The version skew policy that governs how far apart your control plane and kubelets are allowed to drift
  • The correct kubeadm upgrade order: control plane first, then nodes one at a time
  • How EKS, GKE, and AKS change — and simplify — this process on managed control planes

Why This Matters

Kubernetes ships a new minor version roughly every four months, and each minor version is supported for about 14 months. A cluster that never upgrades eventually falls off supported versions entirely, loses access to security patches, and accumulates enough version drift that the eventual forced upgrade becomes far riskier than a series of small ones would have been.

Mental Model

Kubernetes deliberately allows some version drift between components, but only within specific bounds — and the bounds all point in the same direction: the control plane must never be older than the components below it.

The version skew policy

Component pair Maximum skew allowed
kube-apiserver instances (HA cluster) Must all be within 1 minor version of each other
kubelet vs. kube-apiserver kubelet may be up to 3 minor versions older (as of recent policy; historically 2) — but never newer
kube-controller-manager/kube-scheduler vs. kube-apiserver Must not be newer than the API server; may be up to 1 minor version older
kubectl vs. kube-apiserver May be 1 minor version older or newer

The rule that matters most in practice: kubelets can lag behind the API server, but never lead it. This is exactly why upgrades always go control plane first — upgrading a kubelet to a newer minor version than the API server it talks to is unsupported and can break in unpredictable ways.

flowchart LR
    A["v1.36 API server"] -->|supports kubelets down to| B["v1.35, v1.34, v1.33"]
    A -.->|never supports| C["v1.37 kubelet"]

The kubeadm upgrade workflow

Upgrades happen one minor version at a time — you cannot skip from 1.35 straight to 1.37, you go 1.35 → 1.36 → 1.37, verifying health at each step.

0. Point the package repository at the new minor version (every node)

The pkgs.k8s.io repositories are per minor version, so apt can't see 1.37 packages until you change the repository line. Skipping this is the most common reason an upgrade "can't find" the new version:

sudo sed -i 's#/v1.36/#/v1.37/#' /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
apt-cache madison kubeadm | head -3        # confirm 1.37.x packages are now visible

Before touching anything: take an etcd snapshot (Backup and Restore), and check that nothing still uses an API removed in the target version (kubent, pluto, or the API server's apiserver_requested_deprecated_apis metric will tell you).

1. Upgrade the first control-plane node

# On the first control-plane node
sudo apt-mark unhold kubeadm
sudo apt-get install -y kubeadm=1.37.0-1.1
sudo apt-mark hold kubeadm

sudo kubeadm upgrade plan          # shows what will change, sanity-checks skew
sudo kubeadm upgrade apply v1.37.0

sudo apt-mark unhold kubelet kubectl
sudo apt-get install -y kubelet=1.37.0-1.1 kubectl=1.37.0-1.1
sudo apt-mark hold kubelet kubectl
sudo systemctl daemon-reload
sudo systemctl restart kubelet

2. Upgrade remaining control-plane nodes

# On each additional control-plane node
sudo kubeadm upgrade node
# then upgrade kubelet/kubectl packages exactly as above

3. Upgrade worker nodes, one at a time

# From a machine with kubectl access
kubectl drain node-worker-1 --ignore-daemonsets --delete-emptydir-data

# On node-worker-1
sudo kubeadm upgrade node
sudo apt-get install -y kubelet=1.37.0-1.1 kubectl=1.37.0-1.1
sudo systemctl daemon-reload
sudo systemctl restart kubelet

# Back on the kubectl machine
kubectl uncordon node-worker-1
kubectl get nodes   # confirm Ready at the new version before moving to the next node

Repeat the drain → upgrade → uncordon cycle for every worker node, one at a time, confirming health before moving on. This is the same cordon/drain discipline from Node Management applied to an entire fleet.

Never upgrade all nodes simultaneously

Draining and upgrading nodes one at a time is what keeps the application available during the upgrade. Upgrading every node at once is functionally a full-cluster outage.

Before You Upgrade: Checks That Catch Real Outages

Kubernetes doesn't support downgrades. If an upgrade breaks something, the way back is a restore from backup, so the checks happen before, not after:

  • Read the Urgent Upgrade Notes at the top of the target version's changelog. They list what the release removes or changes in a way that can break you.
  • Search workloads and add-on configuration for labels, taints, and fields the release removes. Custom resources count: Reddit's outage hid in Calico's route reflector config, not in any Deployment.

    kubectl get deploy,ds,sts,job -A -o yaml | grep -n "node-role.kubernetes.io/master"
    
  • Ask the API server whether anything still calls deprecated APIs. The counter covers every client since the API server last started, including ones not in your Git repository:

    kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis
    
  • Confirm every node runs cgroup v2 (stat -fc %T /sys/fs/cgroup prints cgroup2fs). Since v1.35 the kubelet refuses to start on cgroup v1 unless failCgroupV1: false is set.

  • Check that the CNI, CSI drivers, ingress or Gateway controller, and service mesh support the target version.
  • Look for PodDisruptionBudgets that will block drains: any PDB showing ALLOWED DISRUPTIONS 0 in kubectl get pdb -A will stall a node.
  • Take an etcd snapshot and know how to restore it (see Backup and Restore).
  • Rehearse on a cluster with the same add-ons. A clean cluster built last week is not the one that will surprise you.

Real incident: Reddit, March 2023

Reddit's upgrade of its oldest, largest cluster from 1.23 to 1.24 removed the node-role.kubernetes.io/master label. Calico's route reflectors were selected by that label, so they vanished, Pod networking collapsed, and Reddit was down for 314 minutes before the team restored from backup. The full story.

As of October 2026 the upstream project supports 1.37, 1.36, and 1.35, and 1.34 reaches end of life on October 27, 2026. A cluster on 1.34 or older is out of upstream patch support unless a vendor extends it.

How Managed Platforms Change This

Platform Control plane upgrade Node upgrade
Self-managed (kubeadm) Fully manual, as above Fully manual, one node at a time
Amazon EKS One eksctl upgrade cluster / console click; AWS handles etcd and API server Managed node groups support rolling upgrade via eksctl upgrade nodegroup; self-managed node groups still need manual draining
Google GKE Automatic by default (configurable maintenance windows); can be manual Node auto-upgrade is on by default, respecting PodDisruptionBudgets during rolling replacement
Azure AKS az aks upgrade triggers a managed, orchestrated control-plane upgrade Node pools upgrade via az aks nodepool upgrade, with configurable surge and max-unavailable settings

The version skew policy still applies underneath all three managed platforms — they just automate the sequencing and safety checks that kubeadm upgrade makes you run by hand. The trade-off is control: managed platforms restrict how far you can defer an upgrade, and deferring can cost money:

  • EKS gives each version about 14 months of standard support, then extended support, which clusters enter automatically unless you opt out. It's billed at $0.60 per cluster per hour instead of $0.10, six times the standard rate, so clusters left on old versions quietly become one of the larger line items on the bill.
  • GKE enrolls clusters in release channels (Rapid, Regular, Stable, Extended) and upgrades them automatically within your maintenance windows and exclusions.
  • AKS offers a paid long-term support option for selected versions.

Budget for a routine upgrade every quarter or so; it's far less work than a forced multi-version jump.

Common Mistakes

  • Upgrading a kubelet to a newer minor version than the control plane — unsupported, and can cause API compatibility failures.
  • Skipping minor versions (1.35 straight to 1.37) instead of stepping through each one.
  • Upgrading all worker nodes at once instead of one at a time with drain/uncordon between each.
  • Forgetting to apt-mark hold/pin the package versions, letting an unrelated apt upgrade silently bump Kubernetes components out of band.
  • Assuming a managed platform's "automatic" node upgrade doesn't need PodDisruptionBudgets — it still respects them, and a misconfigured PDB can still stall a managed upgrade.

Interview Questions

  • What is the Kubernetes version skew policy, and which direction is the drift allowed to go?
  • Why must the control plane always be upgraded before the kubelets?
  • Walk through the full kubeadm upgrade sequence for a 3-control-plane, 5-worker cluster.
  • How does upgrading an EKS-managed node group differ operationally from upgrading self-managed kubeadm worker nodes?

See Interview Prep for full answers.

Next

Continue to Backup and Restore — take a fresh etcd snapshot before every upgrade, so a bad one is always recoverable.