Toil and Automation¶

By Sameer Alam · Updated 14 September 2026

What You'll Learn¶

• What toil is, and how it differs from valuable operational work
• How to measure toil so it can be prioritized
• How to decide what to automate, and in what order
• Patterns that make automation safe, and how self-service removes whole categories of toil
• How DORA metrics show whether delivery and operations are improving

What Toil Is¶

Toil is operational work that is:

Examples: manually rotating certificates, resizing disks when an alert fires, creating accounts or namespaces on request, restarting a service that leaks memory, copying data between environments, approving routine access requests.

Not everything operational is toil. Incident response, postmortems, capacity planning, and designing automation are engineering work. Overhead such as meetings isn't toil either — it's just overhead.

Why It Matters¶

Toil grows with the system. If a team spends most of its time on toil, it can't build the improvements that would reduce it — and as the service grows, the team drowns. Google's SRE practice caps toil at 50% of each engineer's time, and aims well below that, so there's always capacity for engineering work.

Toil also hurts people: repetitive interrupt-driven work is demoralizing, error-prone, and a common reason engineers leave.

Measure It¶

You can't prioritize what you don't count. Pick a lightweight method:

• Ticket labels. Tag operational tickets toil with a category, and record time spent.
• On-call logs. Record every interrupt during the shift: what, how long, and whether it repeats.
• Periodic surveys. Ask engineers each quarter to estimate toil by category.

| Task                                  | Frequency  | Minutes each | Hours / month | Risk if done wrong |
|---------------------------------------|------------|--------------|---------------|--------------------|
| Create namespace + RBAC for new team  | 6 / month  | 45           | 4.5           | Medium             |
| Resize EBS volumes after disk alerts  | 10 / month | 20           | 3.3           | High               |
| Restart report-worker (memory leak)   | 20 / month | 10           | 3.3           | Low                |
| Rotate internal TLS certificates      | 4 / month  | 60           | 4.0           | High               |
| Grant temporary prod DB access        | 30 / month | 15           | 7.5           | High               |

Decide What to Automate¶

Prioritize by time saved, risk reduced, and growth, against the cost of building and maintaining the automation.

quadrantChart
    title Automation priority
    x-axis Low frequency --> High frequency
    y-axis Low risk --> High risk
    quadrant-1 Automate first
    quadrant-2 Automate or add guardrails
    quadrant-3 Leave manual or document
    quadrant-4 Automate when convenient
    Prod DB access: [0.8, 0.85]
    Cert rotation: [0.35, 0.9]
    EBS resize: [0.55, 0.7]
    Namespace creation: [0.45, 0.45]
    Restart worker: [0.75, 0.2]

Also ask whether the task should exist at all:

Levels of Automation¶

Automation doesn't have to be all or nothing:

Move up a level only when the previous one is reliable and well understood.

Make Automation Safe¶

Automation acts fast and at scale — including when it's wrong.

• Dry-run by default, and show exactly what will change.
• Idempotency: running twice produces the same result as running once.
• Limits and circuit breakers: never terminate more than N instances, resize more than N volumes, or change more than N% of a fleet in one run. Stop if error rates rise.
• Rate limiting and staged rollout: one host, then one AZ, then everything.
• Audit logging: record who or what triggered each action, and what changed.
• Least privilege: automation roles can do only what the task needs.
• A kill switch: a simple way to disable the automation during an incident.
• Tests, including against a sandbox — see Testing Python Automation.

MAX_TERMINATIONS_PER_RUN = 5

candidates = find_unhealthy_instances()
if len(candidates) > MAX_TERMINATIONS_PER_RUN:
    alert_humans(
        f"{len(candidates)} instances look unhealthy — more than the safety limit "
        f"of {MAX_TERMINATIONS_PER_RUN}. Not terminating anything automatically."
    )
    raise SystemExit(2)

A large number of "unhealthy" instances is more likely to mean a broken health check or a bad deploy than a coincidence. The safe response is to stop and ask a human.

Self-Service¶

Many requests to operations teams are toil for both sides: the requester waits, and an engineer does something routine. Replace request queues with self-service that has guardrails built in:

This is the core idea behind internal developer platforms: the platform team builds paved roads, and product teams move quickly without filing tickets.

Measuring Delivery and Operations: DORA Metrics¶

The DORA research program identified metrics that together describe software delivery performance:

Throughput and stability improve together in high-performing teams — frequent, small changes are both faster and safer. Reducing toil, automating deployments, and practicing fast rollback move all four metrics in the right direction. Track trends for your own teams rather than comparing against other organizations.

Common Mistakes¶

• Treating toil as "just part of the job" and never measuring it.
• Automating a broken process instead of eliminating the cause.
• Automation with no dry-run, no limits, and no kill switch, which turns a small mistake into a fleet-wide outage.
• Big automation projects that take quarters before saving any time, instead of incremental scripts.
• Automation nobody owns, that breaks silently when an API changes.
• Using DORA metrics to rank teams or individuals, which encourages gaming them.

Interview Questions¶

• What is toil? Give examples of work that is and isn't toil.
• How would you measure toil on a team?
• How do you decide which operational tasks to automate first?
• What safeguards should automation that terminates instances have?
• What are the DORA metrics, and why do throughput and stability improve together?

Next¶

Continue to Capacity Planning and Load Testing.