Avoiding a Red Wedding in Production

A few years ago, we ran into a production problem that fundamentally changed how I think about resilience engineering.