Unpredictable effects of Technical Debt

This article has been written after a conversation with a team leader about the importance of the control on what by purpose we decide not to deliver. With the Technical Debt term, in fact, we are describing a lack of requirements that a team decides to not respect for a series of valid reasons: cost, time, likely, impact… Often TDs (for Technical Debt here on) is massively underestimated and a lot of teams pile them because they never have time to fix them.

However one of the reasons to not leave them unresolved is that they could bring to some unpredictable situations. With this true story, we can bring to life the impacts of TDs in a scenario where you do not expect to meet technical debts: the US Air Force, where the most advanced systems are used by the most trained people on the planet for life critical missions.

At the end of the article you can find some reference/source with all the details of this story (no secrets); here however we will focus only on the relevant topics for our presentation so bear with me if this is not 100% precise and technical, my goal is not to present the accident but the lesson we can learn.

Once upon a time, two F15 strike-eagle air-bomber planes were flying in a true war mission …

The story

The two units (we will call them Red1 and Red2) were planned for a long range mission to release munitions on a target far away from the taking off airport. They have been loaded with a full set of weapons and lateral fuel pods: same configuration for both aircrafts, with different load for left and right wings, following a procedure well known by the ground operators.

After take off, the twoF15s follow the flight plan without any problem and made 2 in-flight refuels before the final strike. After the last refuel, Red2 realized some problem with the right pod, so went to a lower level (different pressure) and everything was fine again. This is not uncommon and the procedure to fix this situation is well known by the pilots.

Close to the target, Red2, differently from the planned mission, was in front of Red1, so he released his munitions in advance and, to facilitate Red1, made an unplanned turn on the left, while Red1 made the turn on the right, as planned.

In the middle of the turn, Red2 started a spin. After some tries to recover, the pilot ejected (in enemy territory) and few days after was safely recovered by friendly forces.

Immediately it was clear that this was not a counter attack, but a true flight accident, so USAF opened an investigation to understand the reasons behind it. This is not the place to discuss the details of the report but let me copy and paste the rewritten story above, adding some notes emerged from the investigation (in italic, my understanding from the report).

REWIND…..

The same story … reviewed after the investigation

The two units (we will call them Red1 and Red2) were planned for a long range mission to release munitions on a target far away from the taking off airport. They have been loaded with a full set of weapons and lateral fuel pods: same configuration for both aircrafts, with different load for left and right wings, following a procedure well known by the ground operators. The mission required two different types of Air2Ground weapons: the primary set (3 bombs) has been placed on the left wing because it was known that the F15 right bay had problems with the failover system (TD1); on the right side instead were mounted the second set (4 different bombs) that has an internal system able to override the eventual bay’s failover. This asymmetrical load (TD2) is pretty common and the manual of the airplane did not report any particular caution (TD3): it is to be noted that F15 without any load is intrinsically asymmetrical (e.g. the gun and ammunitions are on the left of the canopy) (TD4). However in this case the difference was 5 times the basic one (TD5).

After the take off, the two planes follow the flight plan without any problem and made 2 in-flight refuels before the final strike. After the last refuel, Red2 realized some problem with the right pod, so went to a lower level (different pressure) and everything was fine again. This is not uncommon and the procedure to fix this situation is well known by the pilots. Red2 checked after a while the quantities in the pod: the left pod was empty, instead the right one has still 40% of the fuel because there was some malfunction (BUG). This unbalanced weight (TD6) increased the asymmetry of the load.

Close to the target, Red2, differently from the planned mission, was in front of Red1 (TD7a?), so he released his munitions in advance and, to facilitate Red1, made an unplanned turn on the left (TD7b?), while Red1 made the turn on the right, as planned.

F15 manuals defined the limits (known as Controllability Performance Parameters) also for turn inclinations and the pilot could have slightly exceeded them (TD8), as per common practice. However CPPs have been calculated in condition without any load (TD9), and in particular without asymmetrical load (TD10).

In the middle of the turn, Red2 started a spin … (we already know the end)

Learning from the story

This has been a massive series of circumstances where ONE non critical bug and TEN technical debts led to a failure. Let me try to describe these behaviours, removing the context, in terms of Technical Debt: there was a known TD1, but this has been commonly “patched”, because of TD3, creating a new TD2, that together with TD4 brought to an under-estimated TD5.

Lesson 1: Technical Debts could be connected each others, so you should be aware of their interrelated effects.

A not critical BUG created TD6; TD1–6 were still hidden and very likely were exposed by TD7 so that TD8, normally considered a safe practice, in this particular situation brought a massive failure.

Lesson 2: BUGs could increase the risk of (unrelated) Technical Debts.

I have added a “?” to the pair of TD7 because these could not be considered Debts. However if it is OK to adapt to the situations, the decision to break the initial plan was un-informed because of TD9 and in particular of TD10. Also TD8 was considered safe in normal conditions but the sequence TD1–10 was not a normal condition.

Lesson 3: Decisions should consider known and potential Technical Debts.

Eventually, if the plan was followed (and Red2 would have turn on the left) the incident could be avoided, but we cannot consider a Red2 fault and in fact Red2 pilot has been completely discharged of the responsibility for the accident.

Conclusions

Here I have simplified a story that actually lasted seven months and saw USAF making simulations and actual flight tests to understand the true impact of the asymmetrical load on the F15. In particular it was proven that even respecting CPPs limits, in this case would have led to the same end, because USAF proved that the actual CPP in these situations should have been ~20% lower.

USAF requested to the the F15 manufacturer to update manuals and instructions considering the limits of unbalanced load, and also specific new simulations been designed to train the pilots in this situations. USAF also required to fix the problem on the bays that has been removed in the following version of the airplane. So a massive change of the ways of working has been generated by this case thanks to the outcomes of the investigation.

The three lessons mentioned above are the essential learning I took from the analysis of this accident; maybe nothing 100% new, but made absolutely vivid with this true example of an unpredictable (?) failure estimated in around 50M$.

References

Reports on airplane accidents (even if sometimes they are sad examples) could bring a lot of learning in understanding the impact of malfunctions, mistakes, but also the lack of soft soft skills such as collaboration, autonomy and leadership. Aviation industry standard imposes investigation on all accidents (even minor ones) at different levels (Companies, Manufacturers, Authorities) and this approach makes it as one of the most safe transport system. Also, to increase collective knowledge, reports are often freely available.

For this particular accident the official investigation report from USAF can be found here (link). I want to thank the AC Drone youtube channel that explained the accident to non experts (the video is in Italian).

[embed]

Originally published at https://www.linkedin.com.