Diving deeper on operational debt

How about those toil piles?

There are many reasons that operational debt exists. Understanding how the debt came to be is an important step in being able to identify the creation or adoption of such debt going forward. Inspired by Martin Fowler, I believe it’s insightful to do this by using the same dimensions — care and intent. I want to plant a thought in the readers mind. When Fowler discusses these quadrants, the question was not so much about how something came to be in a particular quadrant. The question is — once you know that something is in a certain quadrant — does the quadrant influence how we solve the problem? For Fowler, this was about architectural considerations. In our case, it’s more about process problems and accepting risk. We do care very much about how something got into a quadrant — that indicates process gaps that should be addressed as part of the debt.

Operational debt quadrants

How careful one is can be expressed as a measure of risk. Intentionally making a careless decision means accepting a higher level of risk. In the carefully intentional acceptance of debt, the debt is analyzed and a concrete and reasonable plan enacted to deal with it. In the carelessly intentional example, the debt is acknowledged and converted into short-term toil with brittle signifying the elevated risk.

It is difficult to come up with careful and unintentional debt in the context of technical debt. With operational debt, it seems easy. From the site reliability engineering (SRE) perspective, there are always projects that appear — unannounced — but are nonetheless required to meet business goals and which reflect work into the SRE team’s backlog. This is not optional work, although there may be negotiations around timing. So it must be handled carefully, but it is unintentional from the SRE perspective because it was unknown.

Racial bias in risk perception

It is crucial to assess risk based on the inherent, empirical characteristics of the debt, and not the people that are involved. Think about the difference between these two questions:

was this decision carefully made?

versus

did this person make a careful decision?

This is an area where bias can slip in. Sadly, but perhaps unsurprisingly, people’s opinion of whether risk is responsible or reckless may depend on the risk taker’s race or other characteristics. It is so important to be aware of these kinds of biases, and they are everywhere.

What is the interest payment?

Operational debt as a metaphor is imperfect. When you take on financial debt, you repay the loan through a series of regular payments. Those payments include both money towards the principal amount and an interest payment to the bank. In technical debt, the interest payment is the added complexity and time required to work with section of code containing the debt. There is not necessarily a requirement to pay down the principal, although this is often the subject of much debate. Similarly, with operational debt, the only required payment is interest. And interest is toil (as defined in the SRE context). Straight from Google, toil is:

the kind of work that tends to be manual, repetitive, automatable, tactical, devoid of enduring value, and that scales linearly as a service grows.

I will add that those ingredients — manual and repetitive in particular — are also indicators of elevated risk.

Example: runbooks

Let’s see how this plays out with an example. Here’s a common pool of operational debt: your runbooks are out of date. There are hundreds of microservices that are changing and evolving, with new ones releasing weekly. But there are no methods to ensure ongoing freshness of the operational documentation or to periodically audit them. (I know some will argue that one shouldn’t need runbooks in this day and age and I accept that perspective. But many of us still need them.) What does the debt look like on this runbook entropy problem?

The risk is easy to identify. If there is an outage for one of these services, the responding on-call team is going to potentially have the wrong information. This usually increases the amount of time it takes to repair the problem. This means your MTTR (mean time to repair) is going to go up over time. That is very bad, because when MTTR goes up, availability goes down. That means customers are hurt in the process.

The interest payment happens in three ways.

1. The extra time spent during the outage trying to figure out what the runbook was wrong about in order to fix the problem.
2. An additional, one-off task to fix the runbook after the outage is over. Any decent root cause or retrospective process should identify such an action.
3. You are forced to shift resources and delay another project to fix the runbook from #2. Because it is related to an outage, #2 may carry a statutory and more immutable (and visible) deadline that constrains the SRE manager in ways that affect other projects.

And what of the principal? This example is a difficult problem to solve as it involves processes that span multiple teams. A project to fix it might contain the following elements:

• A process for teams to update runbooks whenever appropriate, preferably automatically
• A rollout plan for the process that includes programs to explain the necessary behavioral changes
• Remediation of existing runbooks to put them in compliance (read: current)
• If using scrum, adding some relevant language to the definition of done.
• Engineer mobility across teams needs to be detected so on-call assignments can be changed.
• Periodic audits where groups of runbooks are reviewed.

One thing is clear, though. Without a project like above, the passage of time is not close to sufficient to ensure the debt is retired. By that, I mean that updating runbooks every time you have a problem will not produce the outcome where the runbooks are all fresh if given enough time.

The toil pile

When you examine the debt backlog at a high level, you have to consider risk, interest, principal, and time. Risk is the most critical element of the rank — if it is high enough, you have to address it. It doesn’t matter if it’s operational debt or not, it must be fixed. For lower risk items, just remember time times Toil, or, (t*T).

Time times toil is the total interest you are going to pay until you fix the underlying problem. For every month that you don’t fix it, you are paying the toil in interest. It’s also helpful to imagine not just toil, but frustrating, onerous toil. The kind that would drive anyone crazy. Double-entering data. Having to repeat a process that randomly breaks until it magically succeeds. Nuisance level alerts that must be handled at the expense of dulling your operational posture.

t*T = the work the SRE team must perform until the problem is fixed

I like to think of tT** as the toil pile***, and each item of debt has an appropriately sized toil pile associated with it.

Cost versus toil quadrants

In my overly simplified view of the world, there is a limit to what an SRE team can achieve in executing large debt reduction programs. There are limits to how large a project an SRE team can take on without needing additional resources, commitments, or funding. With that in mind, let’s move away from the metaphor and shift to the cost of paying off the debt. That means paying the interest while fixing the process and repairing any related data. The debt is retired when the issue has an effective process that doesn’t contain either an elevated risk or an ongoing toil requirement. On the other hand, if the interest is low (small toil pile), do we really care?

High risk problems must be solved promptly and the decisions are about how to react at the appropriate priority and allocate resources. For moderate to lower risk items, the decision is about how much toil we can accept relative to the cost of solving the problem. In general, we need to solve for anything where toil has an excessive growth rate. Here’s a take on the relationship between toil and the cost of remediation in quadrant form:

I look at this chart in a very practical way, and honestly, there’s not much point in discussing anything other than the upper right quadrant. Any good manager can deal with the other three quadrants — they are easy or unimportant. A good SRE team will solve the problems on the bottom row without requiring direction. A good SRE manager will solve the upper row by deferring low toil debt and negotiating cross-team collaborations over high-toil, debt reduction programs.

So what’s the deal with high toil, high cost debt? These are typically problems that hurt on a weekly basis, but the size of the effort to fix them is more than the team can bear. The odds are often stacked against SREs team on these items. Consider these potential issues:

• SRE teams typically have less engineering resource intrinsically available — they must dedicate resources to on-call and reliability responsibilities that are both statutory and unpredictable
• Debt projects compete with other, non-debt based reliability projects for priority
• Debt projects are more complex, requiring cross-team collaboration and commitment of resources
• The toil pile, even when not small from the SRE perspective, is often small relative to the cost of retiring the debt when you add in the efforts of other teams

This last part is really important. From a return on investment perspective, you might decide to just accept the ongoing toil because it’s too hard to solve the underlying problem. Or, more positively, you have to wait for other conditions to make a solution more feasible. In the extreme, you will need to work through executive management to raise visibility based on risk so that you can gain commitments around a collaboration to fix the issue. If that is not possible, the only thing left is to explore any toil reduction strategies you can find to make the stuff you have to do less burdensome.

Conclusion

My goal is not to convince anyone that these concepts should transform their work. I find these concepts helpful in understanding problems from different perspectives. That allows me to develop a deeper understanding and makes it easier to envision potential solutions. The toil pile, for example, may be a visceral metaphor for describing the work, but it is also an important dimension in understanding the problem. Likewise, it becomes easier to identify process gaps, rank debt, and hopefully solve problems when you have a deeper understanding of the problem and the cost of its remedy. That is particularly important when it comes to the most daunting of debts, the high toil, high cost items.