Trace Tickle: Observability’s Funny Footprints

Overview

Observability lets you understand a system from the outside by letting you ask questions about that system without knowing its inner workings. Furthermore, it allows you to troubleshoot and handle novel problems easily, that is, “unknown unknowns”. It also helps you answer the question “Why is this happening?”

Your application must be properly instrumented to ask those questions about your system. That is, the application code must emit signals such as traces, metrics, and logs

Difference between distributed tracing and logging

Logging tools collect timestamped events, such as system errors, user interactions, HTTP response statuses, and other metrics, to help development teams detect system anomalies. Logging provides a static overview of incidents that show developers what happened in the application.

Logs from Spring boot app

In contrast, distributed tracing provides an audit trail that clarifies why an incident occurred by correlating various telemetry data collected throughout a service request’s period.

Traces record the end-to-end ‘journey’ of every user request, from the API Gateway through the entire distributed architecture.

Distributed tracing may use logging and other data collection methods for tracing a specific service request.

Sample Trace Flame Graph

Benefits of Tracing

With distributed tracing, developers can trace or visually follow a request path across different Microservices (Services). This visibility helps

• Troubleshoot errors
• Find latency and performance issues

Accelerate software troubleshooting

With distributed tracing, developers can monitor data that passes through complex paths connecting various microservices and data storage. Using distributed tracing tools, developers can track requests and visualize data propagation paths with precision.

Performance issue monitoring

Developers can solve performance issues promptly and minimize service disruptions. One example is finding Slow Code, Query or API call, etc, and fixing it. Then monitor the fixes.

Reduce time to market

Developers review distributed traces to gain insight that speeds up software development, minimizes development costs, understands user behaviors, and improves market readiness.

Challenges

Distributed tracing has simplified the efforts of developers in diagnosing, debugging, and fixing software issues. The following challenges remain for software teams when choosing tracing tools

Performance impact

Instrumentation is not free. It intercepts an application’s operations and collects (often) a large amount of data, which takes additional CPU and memory. This can have a direct effect on throughput and response time, which can affect the end-user experience with the application.

Manual instrumentation

Some tracing tools require developers to manually instrument their codes to generate the necessary traces. When developers modify codes to trace requests, there are risks of coding errors that affect production releases.

Sampling

If we sample 100% data, all the trace data will be collected, and storage costs will increase. If we sample 10% trace, then only the 10% trace data will be available. With limited sample data, we need additional software troubleshooting approaches to capture major issues that escape the tracing tool. We need to choose a sampling(filter) policy to sample the required traces.

Sampling policy

Data Retention Policy

We need to choose the Trace data retention policy. Because older trace data will become outdated due to frequent code changes. So we need to define an appropriate Trace data retention policy.

Tracing Framework

OpenTelemetry Framework

OpenTelemetry is an open-source project managed by the Cloud Native Computing Foundation (CNCF), which provides a unified set of APIs to collect telemetry data from software systems.

It is an Observability framework and toolkit designed to create and manage telemetry data such as traces, metrics, and logs.

OpenTelemetry is not an observability backend like Jaeger, Prometheus, or other commercial vendors.

OpenTelemetry is focused on the generation, collection, management, and export of telemetry. A major goal of OpenTelemetry is that you can easily instrument your applications or systems, no matter their language, infrastructure, or runtime environment.

Note: The storage and visualization of telemetry is not part of Open Telemetry.

Tracer

A Tracer creates spans containing more information about what is happening for a given operation, such as a request in a service.

Trace Exporters

Send telemetry to the OpenTelemetry Collector to make sure it’s exported correctly. Using the Collector in production environments is a best practice. To visualize your telemetry, export it to a backend such as Jaeger, Zipkin, Prometheus, or a vendor-specific backend

Among exporters, OpenTelemetry Protocol (OTLP) exporters are designed with the OpenTelemetry data model in mind, emitting OTel data without any loss of information.

Traces

Traces give us the big picture of what happens when a request is made to an application.

Traces are essential to understanding the full “path” a request takes in your application. Traces are identified using TraceId

Traces in OpenTelemetry are defined implicitly by their Spans. In particular, a Trace can be thought of as a directed acyclic graph (DAG) of Spans, where the edges between Spans are defined as a parent/child relationship.

Here example Trace made up of 6 Spans

Causal relationships between Spans in a single Trace

      [Span A]  ←←←(the root span)
            |
     +------+------+
     |             |
 [Span B]      [Span C] ←←←(Span C is a `child` of Span A)
     |             |
 [Span D]      +---+-------+
               |           |
           [Span E]    [Span F]

Sometimes it’s easier to visualize Traces with a time axis as in the diagram below:

Temporal relationships between Spans in a single Trace

––|–––––––|–––––––|–––––––|–––––––|–––––––|–––––––|–––––––|–> time

 [Span A···················································]
   [Span B··········································]
      [Span D······································]
    [Span C····················································]
         [Span E·······]        [Span F··]

Spans

A span represents a unit of work or operation. Spans are the building blocks of Traces. In OpenTelemetry, they include the following information:

• Name
• Parent span ID (empty for root spans)
• Start and End Timestamps
• Span Context
• Attributes
• Span Events
• Span Links
• Span Status

Span Context

Span context is an immutable object on every span that contains the following:

• The Trace ID represents the trace that the span is a part of
• The span’s Span ID
• Trace Flags, a binary encoding containing information about the trace
• Trace State, a list of key-value pairs that can carry vendor-specific trace information

Span attributes (Tags)

Span attributes are metadata attached to a span. Attributes are mostly auto-collected. But we can add custom attributes to the spans.

Span Status

Each span has a status. The three possible values are: Unset , Error , andOk

Span Kind

This property is used to identify the type of Span. Here are a few Span kind:

SERVER

A server span represents a synchronous incoming remote call, such as an incoming HTTP request or remote procedure call.

CLIENT

A client span represents a synchronous outgoing remote cal,l such as an outgoing HTTP request or database call. Note that in this context, “synchronous” does not refer to async/await, but to the fact that it is not queued for later processing.

INTERNAL

Internal spans represent operations that do not cross a process boundary. Things like instrumenting a function call or an Express middleware may use internal spans.

Span Links (Links between spans)

A Span may be linked to zero or more other Spans (defined by SpanContext) that are causally related. Links can point to Spans inside a single Trace or across different Traces.

Tracing Architecture

Components

The following are the 3 components involved in collecting & Storing tracing data

1. Application

By using a Java Agent jar file, our application generates Trace data. Then, the generated trace data is exported to OTel Collector.

Java agents are a special type of class that, by using the Java Instrumentation API, can intercept applications running on the JVM, modifying their bytecode.

2. Datadog Agent or OTEL Collector

The Datadog Agent/Collector offers a vendor-agnostic implementation of receiving, processing, and exporting telemetry data. It acts as a middleman between the applications and Datadog.

3. Datadog or Prometheus

Datadog is a SaaS platform that analyzes log, metric, and trace data. All the trace data will be stored in Datadog and available to view, search, export, etc.

Prometheus and Grafana are commonly used together for monitoring and observability. Prometheus is a monitoring and alerting toolkit, while Grafana is a data visualization platform. Prometheus collects metrics from various sources, and Grafana visualizes those metrics in dashboards and graphs

Resources

• https://docs.datadoghq.com/tracing/trace_collection/automatic_instrumentation/dd_libraries/java/?tab=wget
• https://opentelemetry.io/docs/zero-code/java/agent/getting-started/