A Perspective on SRE and Mindshift

[This blog reflects my personal perspective and does not represent the official stance of any company.]

In the rapidly evolving landscape of “everything as a Service,” the importance of ensuring service resiliency has taken center stage. Site Reliability Engineering (SRE), originally introduced by Google’s VP of Engineering Ben Sloss in 2003, has now evolved into a widely practiced methodology.

In my view, SRE embodies a kind of practical wisdom when it comes to delivering or managing a service. Yet, it’s astonishing how often we overlook these common-sense principles. In my various roles, I’ve had to encourage managers to consider Service Level Indicators (SLIs) and have faced challenges in obtaining System Level Objectives (SLOs) for their systems. Incidents were sometimes treated as someone else’s concern.

My journey into online services started at Y! in 2005, where @David Henke led as the SVP of Engineering and Ops. During my time there, I internalized the mantra “Site up is P0,” signifying that site availability takes precedence above all else. @David Filo used to be one of the first person to join IRC incident chatroom.

At Microsoft’s Bing Ads, the leadership team including RDM (@Rajat Taneja, @Don Gagne, and @Munil Shah) meticulously reviewed every critical incident. They consistently upheld availability and quality as core priorities.

During my stint at Visa, particularly with the launch of VDP, every error underwent rigorous analysis, root cause identified, and subsequent resolution. Incidents were reviewed by most of us, including @Nitin Chandel, who was the SVP at the time. Visa operates all its transaction systems at an impressive 6 9’s (99.9999% uptime), a feat that might seem incredible (achieving 6 9’s means experiencing a maximum of only 31.5 seconds of downtime in a year!). Such high availability demands exceptional discipline and automation. Security and resiliency take precedence over features without compromise. Obtaining exceptions to these priorities is nearly impossible, and developers have fully embraced the “shift left” concept.

In my perspective, while technology is undoubtedly pivotal, SRE equally, if not more so, represents a shift in mindset. Pause for a moment and consider that someone else’s business hinges on the availability of your service. This realization underscores the immense responsibility placed on you as a service owner. To assess an organization’s stance on operating services, consider a few simple checkpoints:

• Does the organization have overarching goals related to resiliency?
• Does top management regularly review service performance?
• Are SRE-related requirements and non-functional requirements (NFRs) incorporated into project mandates?
• Does the organization have real-time dashboards accessible to all members?
• Do managers promptly respond to alerts and ensure timely resolution?

By addressing these queries, organizations can gauge their commitment to resiliency and operational excellence. SRE extends beyond adopting advanced technologies; it entails a fundamental shift in perspective. Recognizing the impact your service has on others’ businesses fosters accountability and duty. Embracing a culture of resiliency and adopting practices like operational reviews, integrating SRE requirements, and responsive alert management empowers organizations to develop robust and dependable services in the dynamic digital landscape.

As I mentioned, SRE is grounded in common sense. However, as a friend once remarked, “Common sense is the most uncommon thing.”