LLMOps series — Part 2: Multi-LLM Observability Dashboard (Amazon Bedrock, OLLAMA, OpenAI) with Langfuse

Disclaimer: The content presented in this blog reflects my personal opinions and does not necessarily represent the views or opinions of any organization or entity I am affiliated with.

In previous years, enterprises have been experimenting with Generative AI (GenAI), and now they are gradually moving towards deploying production-ready applications using GenAI. Along with production deployment, there is a requirement for monitoring and alerting based on the performance of the large language model (LLM). Tools are emerging in the LLMOps area. Therefore, I decided to test a few tools to evaluate their ease of configuration and operational insights regarding LLM.

In this blog series, I will be experimenting with three products — Grafana (https://grafana.com) , Langfuse(https://langfuse.com) , and OpenLIT (https://openlit.io).

Refer **part 1**, for the Grafana and its configuration.

In this blog we will discuss about Langfuse.

Use cases:

• Monitor invocations, latencies, logs/traces for multiple LLMs (Amazon Bedrock, OLLAMA (local), Azure OpenAI)
• Manage Prompts

Langfuse is a very widely used Open Source LLM engineering platform.

Step 1: Deploy Langfuse

Langfuse can be deployed online or self-hosted using VM, Kubernetes, or Docker. I used a VM with docker-compose for experimentation.

The deployment should be as simple as cloning the repo and using the Docker compose file. However, version 3 of Langfuse has issues bringing up the environment, so I used version 2 to make it work.

services:
  langfuse-server:
    image: langfuse/langfuse:2
    depends_on:
      db:
        condition: service_healthy
    ports:
      - "3000:3000"
    environment:
      - DATABASE_URL=postgresql://postgres:postgres@db:5432/postgres
      - NEXTAUTH_SECRET=mysecret
      - SALT=mysalt
      - ENCRYPTION_KEY=0000000000000000000000000000000000000000000000000000000000000000
      - NEXTAUTH_URL=http://localhost:3000
      - TELEMETRY_ENABLED=true
      - LANGFUSE_ENABLE_EXPERIMENTAL_FEATURES=false

  db:
    image: postgres
    restart: always
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 3s
      timeout: 3s
      retries: 10
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
      - POSTGRES_DB=postgres
    ports:
      - 5432:5432
    volumes:
      - database_data:/var/lib/postgresql/data

volumes:
  database_data:
    driver: local

Bring up the container using,

docker-compose up -d

Please navigate to http://localhost:3000, register a new account, and then log in using the newly created credentials.

The initial interface prompts you to create a new organization and a project within that organization to begin.

Create your project, go to “Settings” and “API Keys”, then click “Create new API keys”. You’ll need this key to configure your Python app to connect to Langfuse. The key is visible only during creation. Save it safely.

Step 2: Deploy the application

Let's now configure the Python application to use .

Configure the Python environment

First create a .env file to store the credentials,

# Langfuse configuration
LANGFUSE_SECRET_KEY="" # The secret key generated in the above step
LANGFUSE_PUBLIC_KEY="" # The Public key generated in the above step
LANGFUSE_HOST="http://localhost:3000" # The Langfuse host URL

# If you are using Ollama, add these parameters
OLLAMA_ENDPOINT="http://localhost:11434" # Provide your specific configuration
OLLAMA_MODEL_NAME="llama3.2:1b"          # Provide your specific configuration

# If you intend to use Azure OpenAI LLM, add these parameters
OPENAI_API_VERSION=""               # Provide your Azure OpenAI configuration
AZURE_OPENAI_ENDPOINT=""            # Provide your Azure OpenAI configuration
AZURE_OPENAI_API_KEY=""             # Provide your Azure OpenAI configuration
AZIRE_OPENAI_MODEL_NAME="gpt-4-32k" # Provide your Azure OpenAI configuration
AZURE_OPENAI_DEPLOYMENT_NAME=""     # Provide your Azure OpenAI configuration

Install Python packages,

pip install langfuse langchain_ollama langchain_aws langchain_openai --upgrade --quiet

As you can see above, I am using Langchain for configuring the LLM for OLLAMA, Amazon Bedrock , and Azure OpenAI.

Define the Langfuse configuration

Following are the configuration to be enabled for Langfuse,

# Langfuse modules
import os
from langfuse import Langfuse
from langfuse.callback import CallbackHandler

from dotenv import load_dotenv
load_dotenv()

# Initialize Langfuse
langfuse_handler = CallbackHandler()
langfuse_handler.auth_check()
langfuse = Langfuse()

Using Langfuse with Ollama

Following code shows how to use Langfuse with Ollama,

# Import the necessary packages
from langchain_ollama.llms import OllamaLLM
from langchain.prompts import PromptTemplate

# Model definition for Ollama
model = OllamaLLM(
    base_url=os.environ['OLLAMA_ENDPOINT'], 
    model=os.environ['OLLAMA_MODEL_NAME'])

# Define the prompt
prompt="Provide only yes or no. Sun rises in the west."

# Invoke the LLM call
response = model.invoke(
    prompt,
    config={"callbacks":[langfuse_handler]}) # Add Langfuse callback handler

# Print the response
print(response)

Using Langfuse with Amazon Bedrock

Following code shows how to use Langfuse with Amazon Bedrock,

I have already authenticated to Amazon using the credential file, so I am not passing that information here. You will need to configure your respective authentication mechanism.

# Import the necessary packages
from langchain_aws import BedrockLLM

# Model definition for Amazon Bedrock
llm = BedrockLLM(
    model_id="amazon.titan-text-express-v1",
    callbacks=[langfuse_handler] # Add Langfuse callback handler
    )

# Define the prompt
prompt="Provide only yes or no. Sun rises in the west."

# Invoke the LLM call
response = llm.invoke(prompt)

# Print the response
print(response)

Using Lanfuse with Azure OpenAI

Following code shows how to use Langfuse with Azure OpenAI,

# Import the necessary packages
from langchain_openai import AzureChatOpenAI

# Model definition for Azure OpenAI
llm = AzureChatOpenAI(
    name=os.environ["AZIRE_OPENAI_MODEL_NAME"],
    verbose=True,
    temperature=0.34,
    api_key=os.environ["AZURE_OPENAI_API_KEY"],
    openai_api_version=os.environ["OPENAI_API_VERSION"],
    deployment_name=os.environ["AZURE_OPENAI_DEPLOYMENT_NAME"],
    azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
    callbacks=[langfuse_handler] # Add Langfuse callback handler
)

# Define the prompt
prompt="Provide only yes or no. Sun rises in the west."

# Invoke the LLM call
response = llm.invoke(prompt)

# Print the response
print(response.content)

In all the scenarios above, you might have noticed the Lanfuse is attached to the LLM call using the callback feature.

Viewing the Langfuse Dashboard

After running these LLM Invocations, you can see the Traces, Model Costs, and Usage statistics in the Lanfuse Dashboard as shown below,

You may also get additional information and traces about each of the invocations in the Tracing section.

You can define custom cost parameters for each of the models,

Prompt management using Langfuse

Langfuse also provide capability to manage prompts lifecycle. Lets create a prompt in Langfuse and use it in the application.

In the screenshot below, I have provided a simple prompt with least token usage and parameters

After creating the prompt, you can handle the lifecycle of the prompt. For example, I have moved the prompt to production as shown below,

Now you can use this prompt in the application as shown in the Ollama example below,

# Import the necessary packages
from langchain_ollama.llms import OllamaLLM
from langchain.prompts import PromptTemplate

# Model definition for Ollama
model = OllamaLLM(
    base_url=os.environ['OLLAMA_ENDPOINT'], 
    model=os.environ['OLLAMA_MODEL_NAME'])

# Define the prompt using Lanfuse
prompt_template = PromptTemplate.from_template(
    langfuse.get_prompt("sample-prompt").get_langchain_prompt()
)
prompt = prompt_template.format(direction="west")
# Invoke the LLM call
response = model.invoke(
    prompt,
    config={"callbacks":[langfuse_handler]}) # Add Langfuse callback handler

# Print the response
print(response)

Here are some of my findings with Langfuse,

Pros

• Langfuse is designed for LLM observability with pre-made dashboards for Traces, Cost, Invocation history, and dataset management.
• Custom model costs can be added via UI, useful for private deployments and internal chargebacks.
• Numerous native integrations are available.

Cons

• Standard Dashboard customization is not possible, but new dashboards can be requested.
• Version 3 of Langfuse is buggy, possibly due to Clickhouse integration; many users reported issues with the docker-compose setup.
• No alert features for high latency or other conditions.

I will share more details about OpenLIT in my upcoming blog. Stay tuned. Please share your thoughts and expert opinion on LLMOps as a reply to this topic.