How Google SRE Defines Toil

Google introduced the concept of toil

Toil- Manual, repetitive, automatable, tactical work that does not create long-term value.

Toil- Manual, repetitive, automatable, tactical work that does not create long-term value.

Google’s Official Toil Characteristics-

1️⃣ Manual- A human has to do it. Example:- Restarting a service manually Running a script every day

2️⃣ Repetitive- You do the same thing again and again. Example: If it feels like “I’ve done this 100 times” → it’s toil. -Same alert every week -Same disk cleanup every month

3️⃣ Automatable — If it can be automated but isn’t — it’s toil. Example: If automation is possible → but not implemented → it’s toil.

Instead of:Manually creating user accounts You could:Automate via IAM + scripts

4️⃣ Tactical (Not Strategic) It fixes today’s problem but doesn’t improve the system. Example:

Restarting a crashed service (temporary fix) Clearing logs without fixing log rotation

5️⃣ No Long-Term Value- It doesn’t: Improve reliability Improve performance Improve architecture It just keeps the system running.

(Google SRE guideline -Toil Rule ):

Toil should not exceed 50% of an SRE’s time. Meaning: At least 50% of time should be engineering work Automation, reliability improvements, scaling, performance tuning If SREs spend 80–90% in firefighting → it’s unhealthy.

Simple Real-Life Example Imagine: Every Monday: Disk becomes full Engineer clears logs manually

This is toil because: ✅ Manual ✅ Repetitive ✅ Automatable (log rotation) ✅ No long-term fix

Correct SRE approach: Implement log rotation Add disk alert Add auto-cleanup policy Now → toil eliminated.

Toil Measure- In production, toil is measured by tracking the percentage of time spent on repetitive operational work, recurring incident frequency, alert noise ratio, manual interventions, and repeat ticket patterns. The goal is to keep toil under 50% and continuously automate recurring issues.

1) % of Time Spent on Toil Formula: Toil % = (Hours spent on repetitive operational work / Total engineering hours) × 100 If in a month: Total working hours = 160 Incident handling + manual ops = 90 hours Toil % = (90 / 160) × 100 = 56% That’s too high (above 50%).

2 ) Ticket Volume Analysis — Use your ITSM tool (ServiceNow, Jira, etc.) Measure: Number of repetitive tickets Same issue count , Same resolution steps repeated Example: 40% of tickets are “restart service” 25% are “disk full” That’s measurable toil.

3) Alert Noise Ratio Measure: Total alerts generated vs actionable alerts Example: 500 alerts per week Only 50 require action 90% alert noise = hidden toil.

4) Incident Recurrence Rate Measure: Repeat incidents / Total incidents If same issue happens 5 times per month → that’s recurring toil.

5) Mean Time Spent Per Incident (MTTR Pattern) If:MTTR is high, Same troubleshooting steps repeated That signals automation opportunity. You already work in observability, this is where: Monitoring + RCA tracking Post-incident review tagging help identify repeated toil.

6) Deployment Toil Metrics Measure: Manual deployments % Rollback frequency Human approval steps Time spent per release If release requires 10 manual steps → that’s deployment toil.

🎯 Real Enterprise Example (Cloud / Kubernetes) Before improvement: 20 pod restarts per week manually 60% of alerts were false positives 45% tickets were recurring issues

After: Implemented auto-healing Alert threshold tuning Log rotation automation

Result: Toil reduced from 65% → 30% MTTR reduced by 40% Alert noise reduced by 70%