SRE Observability

Observability is a foundational element of SRE. With good observability we will not only find the problem quickly but also get to the root cause quickly. Observability is often compared with Monitoring; but they are not the same. Let’s understand What and how part of Observability in this paper.

Observability — as we all know, has 3 Pillars listed below.

1. Metrics

2. Logs

3. Traces

Let’s deep dive

Metrics:

Metrics will fall under monitoring and cover foundational concepts to monitor an application or Infrastructure component. Metrics have the following sections.

1. Golden Signals: Golden Signals are defined as Traffic, Errors, Saturation and Latency (use this acronym — TESLa). Whenever there is a problem in the application or infrastructure one of the golden signals will be impacted, that’s why they are called Golden Signals.
2. Let’s dig deep.

a. Traffic — Traffic is the main metric but will change depending on application or technology type. For UI application traffic can be pages/hour or Users/hour. For API’s it is transactions/sec (TPS) for any transaction that API support. For MQ’s it will be Queue depth or number of incoming messages.

b. Errors — This signal also depends on application or technology. IT has metrics like 5XX errors or 4XX errors or custom errors or uncaught errors from the code.

c. Saturation — This Signal is important and will be different from application perspective or infrastructure perspective or Network perspective. For EX: if we have Java app running in JVM, we can pick metrics like Process CPU, GC, Heap, Thread Usage, Connection count etc. If it is Server or VM or EC2, we need to check for Host Memory/CPU, for storage we will use disk space, for network we use bandwidth etc.

d. Latency — This signal is mostly common across any type of technology or application and will use Response time as metric. For UI applications, we will have DOM Load and Page load times.

Tooling: Depending on the technology or application type, we need to select the right tool for each signal, once the tool is instrumented, we need to check the correctness of the signal and if the signal is real time or it has lag.

2. SLO/SLI with Error Budget: This concept is very important and normally will be built on top of Golden Signals. SLO is Service level objective, which means if an application is providing a Service for EX: Account Balance, we need to define SLOs like Availability, latency, freshness of data etc. More information about this SLO/SLI and Error Budgets can be found with examples from a separate document (will be provided upon request).

Tooling: Dynatrace/Grafana has a built-in function to create SLOs and dashboards. We can have SLO at different periods like hourly, daily, weekly, monthly, and yearly. Please refer to the following table as an example.

As you can see from the table above, even though our target is high like 99%, as you apply time factor business impact will be huge if you apply for a month Vs Hour. Time is a very important factor for defining SLO targets. When we do Resiliency tests, we need to keep these times in to consideration, for example of region failover if it takes more than 36 seconds, we will violate hourly SLO, but for daily SLO it will still be within target even if it takes 10 min.

3. Health Checks — There are 2 types of Health checks.

a. Shallow Health check — This type of health check is used at app level and Infra level. At app level we can check if we are getting 200, at infra level we can use Port Ping, IP ping etc.

b. Deep Health check — This type of health check is more suitable for applications to check beyond “200 OK” messages. This type of health check validates if data received for a 200 message is correct. For EX: when you pull account details, are we seeing account details data or just blank page with 200 messages.

4. Alerting — Alerting is not part of Metrics, but it is more like a reactive measure taken to identify an issue when it occurs. All Golden Signals should have alerts with an agreed threshold which when breached will cause business impact. Alerts are two types Warning and Alert. Warnings will be used as needed to remove the panicking.

Order of Priority to handle this section of Observability.

1. Golden Signals

2. SLO

3. Health Checks

4. Alerting

Logs:

Logs can be any type of data produced by an application or an infrastructure component. Logs should have enough data to identify details about an error, source, and time stamp. From the application side we should also follow Logging standards for Log statements to have enough information about an error. Logging Standards are documented in a separate document and specific to Technology stack. Most common logs are listed below.

1. Application Logs

2. System Logs

3. Event Logs

4. Custom Logs

Traces:

Traces will be used to find root cause of an issue at a more granular level. There are two types of traces.

1. Stack Traces — This type of tracing will be used at code level to identify an issue at method level without debugging the code. It saves time to find the root cause. Tools like Dynatrace or Splunk have this kind of data either by default or depending on log level details.

2. Distributed traces — Distributed tracing will be used to trace a business transaction as it goes through several applications in the call flow. Normally if all applications are properly instrumented, this view will appear by default (For Ex: transaction flow in DTM) or can be stitched together by using trace ID or Span ID.

Outcome: If we do all activities listed above then we can confidently claim observability.

Let’s see an example: Lets say we have a microservice with 3 APIs serving 10 applications with 3 to 4 apps per API. If we have good monitoring, when there is an error, Golden Signal will indicate the error from monitoring at microservice level, but it will not exactly say which consuming app is impacted by the error as multiple apps consuming same service. With good tracing, we can also find which consuming app is impacted from the error. In this example — Metrics part will identify microservice, logs will identify which consuming app is impacted and finally stack traces will help with which part of the code is causing the error.

Pre-requisites:

1. Application Architecture and Deployment Architecture along with Customer Journeys are Identified and documented in a version controlled system like GitHub.

2. Business goals in terms of Availability, Resiliency and Performance should be identified and documented in version controlled system like GitHub.

3. Data Architecture and Security Architecture should be identified and documented in version controlled system like Github.