Skip to content

Production Readiness Checklist

What You'll Learn

  • The concrete, checkable items a workload should satisfy before it takes real production traffic
  • Why "backups are scheduled" and "backups are tested" are different claims, and only one of them is actually true readiness
  • Where in these docs each checklist item is actually taught, so a gap becomes a reading assignment, not a guess

Why This Matters

"Is this ready for production?" is a question teams answer with a feeling far too often. A checklist turns it into something you can actually verify with a command, forces the same bar on every service regardless of who wrote it, and gives whoever is reviewing a launch something more useful than "looks fine to me."

Mental Model

Treat this as a gate, not a suggestion list. Every unchecked item is a specific, predictable way the workload will fail in production — a missed probe means a bad pod stays in rotation, a missing PDB means a routine node drain becomes an outage, an untested backup means "we have backups" is a belief, not a fact.

The Checklist

Workload configuration

  • Liveness and readiness probes are configured, and readiness fails independently of liveness when a dependency (database, cache) is unavailable. Slow starters have a startup probe. Taught in Probes.
  • Graceful shutdown works: the app handles SIGTERM, finishes in-flight requests within terminationGracePeriodSeconds, and has a short preStop delay so traffic drains before it exits. Taught in Pods.
  • Resource requests and limits are set on every container, derived from measured usage, not guesses. Memory limits are set close to actual peak usage (memory is not compressible — an unbounded container can OOM the node). Taught in Cost Optimization and Cluster Sizing and Capacity Planning.
  • A PodDisruptionBudget exists for every multi-replica Deployment/StatefulSet, so a node drain or cluster upgrade can't take out every replica at once — and it allows at least one disruption, so it can't block maintenance forever. Taught in PodDisruptionBudgets.
  • At least 2 replicas, spread across nodes/zones with topologySpreadConstraints or anti-affinity, for anything that isn't explicitly single-instance-by-design.
  • Images are pinned to an immutable tag or digest (never :latest), come from your own registry or a trusted mirror, and were scanned in CI. Taught in Image and Supply Chain Security.
  • The Pod runs as non-root with allowPrivilegeEscalation: false, dropped capabilities, and a RuntimeDefault seccomp profile, so it's admitted under the Restricted Pod Security Standard. Taught in Pod Security Standards.
# Spot-check: does this Deployment have a matching PDB?
kubectl get deployment payments-api -n payments
kubectl get pdb -n payments -l app=payments-api

Access and network security

  • RBAC follows least privilege — no workload or human identity holds cluster-admin or wildcard verbs/resources it doesn't need. Taught in Security.
  • NetworkPolicies are applied, with a default-deny baseline per namespace and explicit allow rules for required traffic only. Taught in Networking and Security.
  • Secrets are not stored as plain manifests in Git — sourced from a secrets manager, sealed, or otherwise encrypted at rest and in the repo. Taught in Configuration and Packaging.
kubectl auth can-i --list --as=system:serviceaccount:payments:payments-api -n payments
kubectl get networkpolicy -n payments

Observability

  • Metrics, logs, and traces are wired up for the workload — not just "the cluster has Prometheus," but this specific service emits metrics that are actually scraped and dashboarded. Taught in Observability.
  • Alerting exists for the failure modes that matter (error rate, latency, restart count, PVC nearing full) and routes to someone who can act on it, not just a channel nobody watches. Taught in Observability.

Resilience and continuity

  • Backups are scheduled — and separately, tested. A schedule proves nothing about restorability; a completed test restore into a scratch namespace or cluster does. Taught in Disaster Recovery and Backup and Restore.
  • RTO/RPO targets are written down for this workload specifically, not inherited vaguely from "the platform team's DR plan." Taught in Disaster Recovery.
  • A documented upgrade plan exists — for the workload's own rollout strategy (maxSurge/maxUnavailable, canary or blue-green if warranted) and for how it behaves during a cluster/Kubernetes version upgrade. Taught in Cluster Administration.
kubectl get pdb -n payments
kubectl rollout status deployment/payments-api -n payments
# Confirm a real restore has been executed and dated, not just scheduled
velero backup describe payments-hourly --details

How It Works

Run this as a gate at two points: before a new workload's first production deploy, and periodically (quarterly is reasonable) for existing services, since requests drift, RBAC accumulates permissions nobody removes, and NetworkPolicies get "temporarily" loosened and never tightened back. A workload that passes every item above has addressed the failure modes that account for the overwhelming majority of real Kubernetes production incidents — not eliminated risk, but converted it from "unknown" to "understood and accepted."

Common Mistakes

  • Treating "backup job runs successfully" as equivalent to "we can recover from this backup" — they are not the same claim.
  • Adding probes and PDBs to a workload but never re-checking them after a refactor changes startup time or replica count.
  • RBAC and NetworkPolicy reviewed once at launch and never again, while both quietly accumulate scope over the workload's life.
  • Treating this checklist as a one-time launch gate instead of a recurring review — production readiness decays without maintenance.

Interview Questions

  • Walk through what you'd check before approving a new service for production traffic on Kubernetes.
  • Why is a scheduled backup not sufficient evidence of disaster-recovery readiness?
  • What's the actual failure mode of deploying a multi-replica Deployment with no PodDisruptionBudget during a node drain?

See Interview Prep for full answers.

Next

This is the last page in Production Engineering. Continue to AI Workloads to serve LLMs and run AI agents on these clusters, or to OpenShift to see how a major enterprise Kubernetes distribution builds several of these same guarantees into the platform itself.