# Cluster Upgrades

## What You'll Learn

- The version skew policy that governs how far apart your control plane and kubelets are allowed to drift
- The correct `kubeadm upgrade` order: control plane first, then nodes one at a time
- How EKS, GKE, and AKS change — and simplify — this process on managed control planes

## Why This Matters

Kubernetes ships a new minor version roughly every four months, and each minor version is supported for about 14 months. A cluster that never upgrades eventually falls off supported versions entirely, loses access to security patches, and accumulates enough version drift that the eventual forced upgrade becomes far riskier than a series of small ones would have been.

## Mental Model

Kubernetes deliberately allows some version drift between components, but only within specific bounds — and the bounds all point in the same direction: **the control plane must never be older than the components below it.**

### The version skew policy

| Component pair | Maximum skew allowed |
| --- | --- |
| `kube-apiserver` instances (HA cluster) | Must all be within 1 minor version of each other |
| `kubelet` vs. `kube-apiserver` | kubelet may be up to 3 minor versions older (as of recent policy; historically 2) — but never newer |
| `kube-controller-manager`/`kube-scheduler` vs. `kube-apiserver` | Must not be newer than the API server; may be up to 1 minor version older |
| `kubectl` vs. `kube-apiserver` | May be 1 minor version older or newer |

The rule that matters most in practice: **kubelets can lag behind the API server, but never lead it.** This is exactly why upgrades always go control plane first — upgrading a kubelet to a newer minor version than the API server it talks to is unsupported and can break in unpredictable ways.

```
flowchart LR
    A["v1.36 API server"] -->|supports kubelets down to| B["v1.35, v1.34, v1.33"]
    A -.->|never supports| C["v1.37 kubelet"]
```

### The kubeadm upgrade workflow

Upgrades happen one minor version at a time — you cannot skip from 1.35 straight to 1.37, you go 1.35 → 1.36 → 1.37, verifying health at each step.

**0\. Point the package repository at the new minor version (every node)**

The `pkgs.k8s.io` repositories are **per minor version**, so `apt` can't see 1.37 packages until you change the repository line. Skipping this is the most common reason an upgrade "can't find" the new version:

```bash
sudo sed -i 's#/v1.36/#/v1.37/#' /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
apt-cache madison kubeadm | head -3        # confirm 1.37.x packages are now visible
```

Before touching anything: take an etcd snapshot ([Backup and Restore](<https://computecentral.in/kubernetes/cluster-administration/05-backup-and-restore/index.md>)), and check that nothing still uses an API removed in the target version (`kubent`, `pluto`, or the API server's `apiserver_requested_deprecated_apis` metric will tell you).

**1\. Upgrade the first control-plane node**

```bash
# On the first control-plane node
sudo apt-mark unhold kubeadm
sudo apt-get install -y kubeadm=1.37.0-1.1
sudo apt-mark hold kubeadm

sudo kubeadm upgrade plan          # shows what will change, sanity-checks skew
sudo kubeadm upgrade apply v1.37.0

sudo apt-mark unhold kubelet kubectl
sudo apt-get install -y kubelet=1.37.0-1.1 kubectl=1.37.0-1.1
sudo apt-mark hold kubelet kubectl
sudo systemctl daemon-reload
sudo systemctl restart kubelet
```

**2\. Upgrade remaining control-plane nodes**

```bash
# On each additional control-plane node
sudo kubeadm upgrade node
# then upgrade kubelet/kubectl packages exactly as above
```

**3\. Upgrade worker nodes, one at a time**

```bash
# From a machine with kubectl access
kubectl drain node-worker-1 --ignore-daemonsets --delete-emptydir-data

# On node-worker-1
sudo kubeadm upgrade node
sudo apt-get install -y kubelet=1.37.0-1.1 kubectl=1.37.0-1.1
sudo systemctl daemon-reload
sudo systemctl restart kubelet

# Back on the kubectl machine
kubectl uncordon node-worker-1
kubectl get nodes   # confirm Ready at the new version before moving to the next node
```

Repeat the drain → upgrade → uncordon cycle for every worker node, one at a time, confirming health before moving on. This is the same cordon/drain discipline from [Node Management](<https://computecentral.in/kubernetes/cluster-administration/03-node-management/index.md>) applied to an entire fleet.

> [!WARNING]
>
> **Never upgrade all nodes simultaneously**
>
> Draining and upgrading nodes one at a time is what keeps the application available during the upgrade. Upgrading every node at once is functionally a full-cluster outage.

## Before You Upgrade: Checks That Catch Real Outages

Kubernetes doesn't support downgrades. If an upgrade breaks something, the way back is a restore from backup, so the checks happen before, not after:

- [ ] Read the **Urgent Upgrade Notes** at the top of the target version's changelog. They list what the release removes or changes in a way that can break you.
- [ ] Search workloads **and** add-on configuration for labels, taints, and fields the release removes. Custom resources count: Reddit's outage hid in Calico's route reflector config, not in any Deployment.

  ```bash
  kubectl get deploy,ds,sts,job -A -o yaml | grep -n "node-role.kubernetes.io/master"
  ```
- [ ] Ask the API server whether anything still calls deprecated APIs. The counter covers every client since the API server last started, including ones not in your Git repository:

  ```bash
  kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis
  ```
- [ ] Confirm every node runs cgroup v2 (`stat -fc %T /sys/fs/cgroup` prints `cgroup2fs`). Since v1.35 the kubelet refuses to start on cgroup v1 unless `failCgroupV1: false` is set.
- [ ] Check that the CNI, CSI drivers, ingress or Gateway controller, and service mesh support the target version.
- [ ] Look for PodDisruptionBudgets that will block drains: any PDB showing `ALLOWED DISRUPTIONS 0` in `kubectl get pdb -A` will stall a node.
- [ ] Take an etcd snapshot and know how to restore it (see [Backup and Restore](<https://computecentral.in/kubernetes/cluster-administration/05-backup-and-restore/index.md>)).
- [ ] Rehearse on a cluster with the same add-ons. A clean cluster built last week is not the one that will surprise you.

> [!NOTE]
>
> **Real incident: Reddit, March 2023**
>
> Reddit's upgrade of its oldest, largest cluster from 1.23 to 1.24 removed the `node-role.kubernetes.io/master` label. Calico's route reflectors were selected by that label, so they vanished, Pod networking collapsed, and Reddit was down for 314 minutes before the team restored from backup. [The full story](<https://computecentral.in/kubernetes/case-studies/07-real-world-outages-and-lessons/index.md#reddit-2023-a-removed-node-label-took-down-pod-networking>).

As of October 2026 the upstream project supports 1.37, 1.36, and 1.35, and 1.34 reaches end of life on October 27, 2026. A cluster on 1.34 or older is out of upstream patch support unless a vendor extends it.

## How Managed Platforms Change This

| Platform | Control plane upgrade | Node upgrade |
| --- | --- | --- |
| Self-managed (`kubeadm`) | Fully manual, as above | Fully manual, one node at a time |
| Amazon EKS | One `eksctl upgrade cluster` / console click; AWS handles etcd and API server | Managed node groups support rolling upgrade via `eksctl upgrade nodegroup`; self-managed node groups still need manual draining |
| Google GKE | Automatic by default (configurable maintenance windows); can be manual | Node auto-upgrade is on by default, respecting `PodDisruptionBudget`s during rolling replacement |
| Azure AKS | `az aks upgrade` triggers a managed, orchestrated control-plane upgrade | Node pools upgrade via `az aks nodepool upgrade`, with configurable surge and max-unavailable settings |

The version skew policy still applies underneath all three managed platforms — they just automate the sequencing and safety checks that `kubeadm upgrade` makes you run by hand. The trade-off is control: managed platforms restrict how far you can defer an upgrade, and deferring can cost money:

- **EKS** gives each version about 14 months of standard support, then **extended support**, which clusters enter automatically unless you opt out. It's billed at $0.60 per cluster per hour instead of $0.10, six times the standard rate, so clusters left on old versions quietly become one of the larger line items on the bill.
- **GKE** enrolls clusters in **release channels** (Rapid, Regular, Stable, Extended) and upgrades them automatically within your maintenance windows and exclusions.
- **AKS** offers a paid **long-term support** option for selected versions.

Budget for a routine upgrade every quarter or so; it's far less work than a forced multi-version jump.

## Common Mistakes

- Upgrading a kubelet to a newer minor version than the control plane — unsupported, and can cause API compatibility failures.
- Skipping minor versions (1.35 straight to 1.37) instead of stepping through each one.
- Upgrading all worker nodes at once instead of one at a time with drain/uncordon between each.
- Forgetting to `apt-mark hold`/pin the package versions, letting an unrelated `apt upgrade` silently bump Kubernetes components out of band.
- Assuming a managed platform's "automatic" node upgrade doesn't need `PodDisruptionBudget`s — it still respects them, and a misconfigured PDB can still stall a managed upgrade.

## Interview Questions

- What is the Kubernetes version skew policy, and which direction is the drift allowed to go?
- Why must the control plane always be upgraded before the kubelets?
- Walk through the full `kubeadm upgrade` sequence for a 3-control-plane, 5-worker cluster.
- How does upgrading an EKS-managed node group differ operationally from upgrading self-managed `kubeadm` worker nodes?

See [Interview Prep](<https://computecentral.in/kubernetes/interview-prep/index.md>) for full answers.

## Next

Continue to [Backup and Restore](<https://computecentral.in/kubernetes/cluster-administration/05-backup-and-restore/index.md>) — take a fresh etcd snapshot before every upgrade, so a bad one is always recoverable.
