How to approach rollbacks during production failures — Real-Time Solutions

Interview Question and Answer

➤ Production failures can occur for many reasons such as code defects, configuration mistakes, database issues, infrastructure problems, third-party dependency failures, traffic spikes, security misconfigurations, or deployment errors.

➤ When a production failure happens, my primary goal is to restore service quickly and minimize user impact, not to immediately debug the issue. I follow a structured rollback-first approach.

➤ First, I detect and confirm the failure by checking alerts, logs, and key metrics like error rate, latency, and resource usage. I also verify whether the issue started after a recent deployment to determine if it’s release-related or caused by infrastructure or external dependencies.

➤ Second, I contain the blast radius. I pause ongoing deployments, reduce traffic if necessary, and disable feature flags related to the change. This helps limit user impact and sometimes avoids a full rollback.

➤ Third, I decide whether to rollback or fix forward. If the impact is high or the fix is not obvious within a few minutes, I choose rollback. In production, stability and availability are more important than fast debugging.

➤ Fourth, I execute the rollback safely based on the deployment strategy. With blue-green or canary deployments, I redirect traffic to the previous stable version. With rolling deployments, I redeploy the last known good artifact and ensure that configuration and environment variables are also reverted. I also verify database compatibility to avoid schema-related issues.

➤ Fifth, I validate the system after rollback. I monitor metrics, review logs, and perform quick smoke tests to confirm that critical user flows are working and the system has stabilized.

➤ Finally, once the system is stable, I conduct a root cause analysis. I identify whether the failure was due to code, configuration, database changes, infrastructure, external services, traffic spikes, security settings, or deployment issues. Based on this, I add missing tests, improve monitoring and alerts, introduce feature flags or gradual rollouts, and update runbooks to prevent the issue from happening again.

Overall, my approach is to detect quickly, limit impact, rollback safely, verify stability, and then strengthen the system so the same type of production failure does not recur.