4 Observability functionality in Spring AI (Milestone 2)

Spring AI 1.0.0 M2 Key Focus: Observability

**Spring AI is one of the most exciting projects in the Spring Framework portfolio. It helps Spring developers create AI-powered applications and integrate Spring-based applications with AI models easily. Spring AI is heavily under development, and by releasing version 1.0.0 M2, it is one step closer to becoming production-ready (version 1). The key focus in Spring AI 1.0.0 M2 was the adding observability functionality to this framework. In this article, we will look at the observability features that were added to the Spring AI 1.0.0 M2**.

🟧 Not a Medium member? Read this article for free.

[embed]🤖 Tutorial: Spring AI, OpenAI, Llama and RAG This series of articles will teach you how to implement an Employee Assistance chatbot using Spring AI and the RAG…zarinfam.medium.com

Spring AI 1.0.0 M2 Key Focus: Observability

· A short introduction to Spring AI ∘ Why is Observability important for Spring AI? · Observability in Spring AI ∘ Enabling Observability functionality in Spring AI ∘ Using Digma to analyze observability data · 4 Observability functionality in Spring AI ∘ 1- Providing observability functionalities for core components ∘ 2- Introducing several metrics related to Spring AI and AI Models ∘ 3- Providing distributed tracing data ∘ 4- Exposing input and output data in the observations · Final Thoughts

[embed]🍃🤖 Spring AI 1.0 — Tour & Feature Set

A short introduction to Spring AI

I am writing a series of articles about Spring AI and implementing a practical project using it, so I want to save time on this introduction. If you are unfamiliar with Spring AI, I recommend reading the **introduction part** of the first part of the series.

To make it short,

Spring AI provides portable APIs and abstractions for AI Model types and providers, Vector databases, and more on top of the Spring Framework concepts.

It offers a feature set that makes developing AI-powered applications much easier.

Why is Observability important for Spring AI?

I wrote a detailed article about the importance of observability in this article:

[embed]Why is OpenTelemetry important for Java applications? The Crucial Role of OpenTelemetry in Modern Java Application Monitoringitnext.io

Spring AI is no exception to this rule, By having different metrics when implementing AI-powered applications, we can have more control over our application. For example, by having metrics for sent tokens to the AI Model, we can measure the usage and prevent additional costs occurs.

On the other hand, Tracing can show us the flow of data and calls between our application components and the AI Model.

Observability in Spring AI

Spring Framework supports observability through **Micrometer for metrics and [Micrometer Tracing](https://micrometer.io/docs/tracing)** for distributed tracing. Similar to other projects, libraries, and frameworks on top of Spring Framework, Spring AI builds upon these observability features in the Spring ecosystem to provide metrics and tracing for AI-related operations in our Spring Application.

Enabling Observability functionality in Spring AI

Similar to a typical Spring Boot project, in order to enable the observability functionality and observing metrics and tracing, we need to add a few dependencies to our project.

• Adding Spring Boot Actuator

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>

• Adding a Tracer Implementation

        <dependency>
            <groupId>io.micrometer</groupId>
            <artifactId>micrometer-tracing-bridge-otel</artifactId>
        </dependency>

• Adding an exporter to store the traces

        <dependency>
            <groupId>io.opentelemetry</groupId>
            <artifactId>opentelemetry-exporter-otlp</artifactId>
        </dependency>

Using Digma to analyze observability data

During the introduction of new observability functionality in Spring AI, instead of using Decker-Compose and using Zipkin or Prometheus container in it, we will use the **Digma plugin for [IntelliJ](https://plugins.jetbrains.com/plugin/19470-digma-continuous-feedback)**.

**Digma is a Continuous Feedback platform that dynamically analyses our code and helps us find performance issues. Besides this, since Digma uses [OpenTelemetry](https://opentelemetry.io/)** behind the scenes, we can configure its observability mode to Micrometer and use it to see metrics and traces generated by our Spring Boot application inside the IDE using the Digma IntelliJ plugin.

4 Observability functionality in Spring AI

As I mentioned before, the main focus of the Spring AI 1.0.0 milestone 2 was adding observability functionality in Spring AI. I have created a branch called [**spring-ai-observability](https://github.com/zarinfam/employee-chatbot/tree/spring-ai-observability) in the Employee Chatbot project GitHub repository that contains all configurations needed to start the chatbot and see the metrics and tracing** in the Digma IntelliJ plugin. The only things that you need to consider in your local development environment are:

• Create an **OpenAI API key** and define it as an environment variable named OPENAI-API-KEY.
• Install the Digma intelliJ plugin and change its Spring Boot observability mode to Micrometer.

1- Providing observability functionalities for core components

Based on the observability features in the Spring framework (Micrometer), Spring AI provides metrics and tracing functionality for its core components, including ChatClient, Advisors, ChatModel, EmbeddingModel, ImageModel, and VectorStore.

For this release, Spring AI only supports observability for OpenAI, Ollama, Mistral, and Anthropic Chat Models implementations. However, They announced that support for other Chat Models implementations will be added to the upcoming versions.

2- Introducing several metrics related to Spring AI and AI Models

Spring AI introduces various useful metrics as low cardinality keys in core components. You can read the complete list for each component in the Spring AI official documents, But we mention some of the important ones.

First, run the Employee Chatbot project in IntelliJ IDE, and then ask some questions like these inside your browser:

http://localhost:8080/employee/chat/1/my%20name%20is%20Deli

http://localhost:8080/employee/chat/1/What%20is%20my%20name?

It is worth mentioning that Digma automatically changes the project run configuration and adds some additional parameters that enable it to collect observability data.

Now, if you go to the actuator URL:

http://localhost:8080/actuator/metrics 

You can see at least 7 more metrics there related to Spring AI:

{
    "names": [
.
.
.
        "gen_ai.client.operation",
        "gen_ai.client.operation.active",
        "gen_ai.client.token.usage",
.
.
.
        "spring.ai.chat.client.advisor",
        "spring.ai.chat.client.advisor.active",
        "spring.ai.chat.client.operation",
        "spring.ai.chat.client.operation.active",
.
.
.
    ]
}

For each metric, there are several tags. For example, if you open the URL for the gen_ai.client.token.usage metric:

http://localhost:8080/actuator/metrics/gen_ai.client.token.usage

We can see the Measures number of input and output tokens used and their available tags:

{
    "name": "gen_ai.client.token.usage",
    "description": "Measures number of input and output tokens used",
    "measurements": [
        {
            "statistic": "COUNT",
            "value": 146
        }
    ],
    "availableTags": [
        {
            "tag": "gen_ai.operation.name",
            "values": [
                "chat"
            ]
        },
        {
            "tag": "gen_ai.response.model",
            "values": [
                "gpt-4o-2024-05-13"
            ]
        },
        {
            "tag": "gen_ai.request.model",
            "values": [
                "gpt-4o"
            ]
        },
        {
            "tag": "gen_ai.token.type",
            "values": [
                "output",
                "input",
                "total"
            ]
        },
        {
            "tag": "gen_ai.system",
            "values": [
                "openai"
            ]
        }
    ]
}

On the other hand, if we check the Digma Observability view, we can see that Digma detected our calling for the Employee Chatbot URL and provided us with a lot of helpful information.

Digma Observability view

3- Providing distributed tracing data

Spring AI provides tracing data for its core components. Several High and low-cardinality keys (tags) will be added to traces. You can check the list of keys for each component from this link.

Let’s check the collected tracing data by Digma. If you click on the Trace button in each row (Assets) in the Digma Observability view, You will see the trace timeline view for that request:

Digma Trace view

As you can see, for each interaction between Spring AI components, such as ChatClient, Advisor, or ChatModel, we have a span. Other than that, we have one span with details for calling the Chat Model implementation (in our case, OpenAI API).

By clicking on each span, we can see all the tags that Spring AI adds. There is also some information about the time spent on each interaction.

4- Exposing input and output data in the observations

We can configure Spring AI ChatClient, ChatModel, ImageModel, and VectorStore components to include input, prompt, completion, or query response in an observation as span attributes. Although these data are usually too big, storing them as span attributes does not make sense. According to the Spring AI documentation, this is because of this limitation:

These data are typically too big to be included in an observation as span attributes. The preferred way to store large data it is as span events, which are supported by OpenTelemetry but not yet surfaced through the Micrometer APIs. Spring AI supports storing these fields as events in OpenTelemetry and will provide a more general event based solution once the issue github.com/micrometer-metrics/micrometer/issues/5238 is resolved.

This feature for all supported components is disabled by default, We can enable them by these configs:

spring.ai.chat.client.observations.include-input=true
spring.ai.chat.observations.include-prompt==true
spring.ai.chat.observations.include-completion=true
spring.ai.image.observations.include-prompt=true
spring.ai.vectorstore.observations.include-query-response=true

For example, if we enable the ChatClient input data as a span attribute. We will see it in the Digma trace view:

Trace’s span details in Digma

Final Thoughts

Observability is an integral part of any library and framework that wants to be used in today’s modern services, and Spring AI is no exception. Adding these observability functionalities to version 1.0.0-m2 of this project has been a big step closer to the final release of Spring AI framework version 1. As you can see, throughout the article, we used the Digma plugin for IntelliJ, which made it very easy for us to view and review the observability data.

[embed]🤖 Tutorial: Spring AI, OpenAI, Llama and RAG This series of articles will teach you how to implement an Employee Assistance chatbot using Spring AI and the RAG…zarinfam.medium.com

🙏 Thanks for reading. You can connect with me on:

**🖊️ Medium | [🐦 Twitter](https://twitter.com/zarinfam)**