# Jobs and CronJobs

## What You'll Learn

- Why a Deployment is the wrong tool for run-to-completion work, and what a Job does instead
- The Job fields that control how much work runs, how much runs at once, and how many failures are tolerated
- CronJob scheduling syntax and the settings that prevent overlapping or pileup runs

## Why This Matters

A Deployment's entire model is "keep N replicas running forever" — feed it a container that's supposed to run once and exit, and it will restart that container forever, which is exactly wrong for batch work. Jobs and CronJobs exist because "run this to completion" and "run this on a schedule" are genuinely different reconciliation problems from "keep this running."

## Job: Run to Completion

```yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: db-migration
spec:
  completions: 1
  parallelism: 1
  backoffLimit: 3
  activeDeadlineSeconds: 900        # give up after 15 minutes, whatever the retries
  ttlSecondsAfterFinished: 86400    # delete the Job and its Pods a day after it finishes
  template:
    spec:
      restartPolicy: OnFailure
      containers:
        - name: migrate
          image: myapp-migrate:1.4.2
          command: ["./migrate.sh"]
```

| Field | Controls |
| --- | --- |
| `completions` | How many successful Pod completions satisfy the Job overall — `1` for a single task, higher for a fixed batch of independent units of work |
| `parallelism` | How many Pods can run at once while working toward `completions` |
| `backoffLimit` | How many times a failing Pod is retried before the Job itself is marked `Failed` (with an exponential backoff between retries) |
| `restartPolicy` | Must be `OnFailure` or `Never` for a Job's Pod template — `Always` (the Deployment default) isn't valid here, since it would fight the whole "run to completion" model |
| `activeDeadlineSeconds` | A hard wall-clock limit for the whole Job; a hung migration fails instead of running forever |
| `ttlSecondsAfterFinished` | Automatic cleanup of finished Jobs and their Pods, so they don't pile up |

```bash
kubectl create job one-off-task --image=busybox:1.36 -- sh -c "echo done"
kubectl get jobs
kubectl logs job/one-off-task
```

A Job with `completions: 5, parallelism: 2` runs at most 2 Pods concurrently until 5 have succeeded in total — useful for a fixed amount of independent, parallelizable work like processing a batch of files.

## CronJob: Jobs on a Schedule

```yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-cleanup
spec:
  schedule: "0 2 * * *"
  timeZone: "Europe/London"         # without this, the schedule uses the controller's time zone (usually UTC)
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 300
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 1
  jobTemplate:
    spec:
      backoffLimit: 2
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: cleanup
              image: myapp-cleanup:1.2.0
              command: ["./cleanup.sh"]
```

A CronJob is a template that creates a new Job object on the schedule you give it — `schedule` uses standard five-field cron syntax (minute, hour, day-of-month, month, day-of-week; `0 2 * * *` means 2 a.m. daily).

`timeZone` takes an IANA name such as `Asia/Kolkata` or `America/New_York`. Without it, "2 a.m." means 2 a.m. in the kube-controller-manager's time zone, which on managed clusters is almost always UTC — a classic reason a "nightly" job runs in the middle of someone's working day. With a real time zone, daylight-saving shifts are handled for you, but a job scheduled inside the skipped or repeated hour may run zero or two times on those days.

| Field | Controls |
| --- | --- |
| `concurrencyPolicy` | What happens if the previous scheduled run is still going when the next one is due — `Allow` (default, runs both), `Forbid` (skip the new run), `Replace` (kill the old run, start the new one) |
| `startingDeadlineSeconds` | How late a missed run can start and still be considered on-time — past this, the run is skipped and counted as missed, rather than run late |
| `successfulJobsHistoryLimit` / `failedJobsHistoryLimit` | How many completed Job objects to keep around for inspection before garbage collection |

```bash
kubectl get cronjobs
kubectl get jobs --watch                    # watch new Jobs appear as the schedule fires
kubectl create job --from=cronjob/nightly-cleanup manual-run-now   # trigger one run immediately, outside the schedule
```

## When a Job Fails: BackoffLimitExceeded

```text
$ kubectl describe job db-migration
...
Conditions:
  Type    Status  Reason
  ----    ------  ------
  Failed  True    BackoffLimitExceeded
```

The Job's Pods failed more times than `backoffLimit` allows (6 if you don't set it), so the Job controller stopped retrying and marked the Job `Failed`. The Job object is fine. The question is why its Pods failed:

```bash
kubectl get pods -l job-name=db-migration
kubectl logs job/db-migration              # logs from one of the Job's Pods
kubectl describe pod <failed-pod>          # exit code, OOMKilled, image pull errors
```

- With `restartPolicy: Never`, every failure leaves a failed Pod behind to inspect. With `OnFailure`, the container restarts inside the same Pod, so check `kubectl logs --previous`.
- A Job stopped by `activeDeadlineSeconds` shows the reason `DeadlineExceeded` instead.
- A failed Job doesn't restart. Fix the cause, then delete and recreate the Job. For a CronJob, trigger a fresh run with `kubectl create job --from=cronjob/<name>`.

## Use Cases

- **Job**: database migrations run once per deploy, one-off data backfills, batch image/video processing split across parallel workers, CI-triggered test runs.
- **CronJob**: nightly backups, periodic cache warming, scheduled report generation, certificate renewal checks, cleaning up expired records.

## What Most Engineers Miss

- **More than 100 missed schedules stops a CronJob cold.** The controller counts the schedules missed since the last run; past 100 it doesn't start the Job at all and logs `too many missed start times`. A CronJob that was suspended, or whose controller was down, for a while can need `startingDeadlineSeconds` set before it recovers, because the deadline limits how far back the controller counts.
- **Your job can run twice, or not at all.** The docs say a CronJob creates a Job approximately once per schedule, and in some situations two Jobs or none; even a Job with one completion and `restartPolicy: Never` may occasionally start its program twice. Make every job safe to run again: idempotent writes, a lock, or a "done" marker.
- **History is short.** CronJobs keep only the last 3 successful and the last **1** failed Job by default (`successfulJobsHistoryLimit`, `failedJobsHistoryLimit`). The failure before last, and its Pod logs, is already gone.
- **`CRON_TZ=` in the schedule is rejected.** Kubernetes fails validation for `TZ` or `CRON_TZ` inside `.spec.schedule`; the `timeZone` field shown above is the supported way.
- **Not every failure deserves a retry.** `backoffLimit` retries any failure, with a growing delay between attempts. A `podFailurePolicy` can fail the Job immediately on an exit code that means "bad input", instead of retrying it six times.

## Common Mistakes

- Using a Deployment for a task meant to run once and exit — it will restart the container in a loop forever, since a Deployment has no concept of "done."
- Setting `concurrencyPolicy: Allow` (the default) for a job that isn't safe to run twice concurrently — e.g., two overlapping backup jobs writing to the same location.
- Forgetting `restartPolicy: OnFailure` or `Never` on a Job's Pod template — leaving the field unset does not default to something Job-compatible in every context, and `Always` is rejected.
- Not setting `successfulJobsHistoryLimit`/`failedJobsHistoryLimit` (for CronJobs) or `ttlSecondsAfterFinished` (for one-off Jobs), letting finished Job and Pod objects accumulate.
- Leaving out `timeZone` and scheduling a CronJob in local time when the cluster runs on UTC.
- A retrying Job that isn't idempotent: with `backoffLimit: 3`, a migration that half-applied before crashing runs up to three more times. Make the work safe to repeat, or use `backoffLimit: 0` and handle failure by hand.

## Check Yourself

> [!NOTE]
>
> **The CronJob that won't start**
>
> A CronJob scheduled every minute was suspended for three hours. After you resume it, it never starts, and the controller logs `too many missed start times`. What fixes it?
>
> - [ ] Delete and recreate the CronJob's Pods
> - [x] Set `startingDeadlineSeconds`, so the controller only counts recently missed schedules
> - [ ] Change `concurrencyPolicy` to `Replace`
>
> With no deadline, the controller counts every miss since the last run, 180 here. Past 100 it refuses to start. A deadline such as 200 seconds bounds the count.

> [!NOTE]
>
> **Exactly once?**
>
> A CronJob sends a monthly invoice email. What guarantee does Kubernetes give about how often each scheduled run happens?
>
> - [ ] Exactly once per schedule
> - [x] Approximately once: occasionally twice or not at all, so the job must be safe to repeat
> - [ ] At least once, never more
>
> Design scheduled work to be idempotent, for example by recording which month's invoices were already sent.

## Interview Questions

- Why can't you just use a Deployment for a batch task, mechanically?
- What's the difference between `completions` and `parallelism` on a Job?
- What does `concurrencyPolicy: Forbid` protect against on a CronJob, and when would `Replace` be more appropriate instead?

See [Interview Prep](<https://computecentral.in/kubernetes/interview-prep/index.md>) for full answers.

## Next

This closes out Core Concepts. Continue to [Workloads and Scheduling](<https://computecentral.in/kubernetes/workloads-and-scheduling/index.md>) for DaemonSets, StatefulSets, deployment strategies, autoscaling, and scheduling controls like affinity and taints/tolerations.
