AI Anomaly Detector, Powered by Flink and Streamlit

courtesy gemini

Lately I have been hearing a lot about how the AI could be integrated in the Data engineering World. Specifically in Realtime use cases.

Considering the Pre AI era, we have used real-time Services like Spark and Flink with Kafka to analyze and act on the streaming events.

We apply aggregation, we process them in the processing layer and we feed it to a downstream job or a dashboard.

In downstream process, you would end up creating another process to analyze this data further. You may want to categorize this data based on the Z score, if its a expected change or an anomaly.

A z-score tells you how many standard deviations a specific data point is away from the mean

Now with AI, you could actually push all the complexity to the initial processing layer and keeping the downstream as light as possible.

We can process the events in real-time, apply the transformation and provide the Z score. Using AI, you could turn the Z score into a Human readable analysis and flag them.

Everything in real-time. No batch, no delay or no fatigue for the human who is analyzing this data and loss to the business.

In this blog, we are going to work towards a Real time solution where we connect Spark and Flink together.

In the end we have our AI Agent talking to our Real-time Pipeline and provide the insights, which earlier could have been very difficult.

Problem Statement

Say we have a Marketplace where we track the price changes.

Mostly we run a batch job which runs either hourly or Daily.

You also need a process to analyze the data if the change is expected or its an anomaly. Its humanly impossible to validate each of the price changes and give a Z score to categorize them.

By the time you process, analyze and flag, its already too late. The damage is already done.

We want to setup a real-time system that watches every price change on a marketplace and catches the bad ones before they cause the damage. Also remember, we want to remove the fatigue where a human is going through this change and flagging them manually.

courtesy gemini

Building the Solution, Step by Step

Event Streaming

We will use Kafka for real-time event streaming, with each of the price changes.

Flink Process

In realtime we read the events, and keeps a rolling stats per product, and compares with with the category baseline using the z-score with the % change.

If at any time, the price change is outside normal behavior, it gets flagged immediately.

Spark batch job

Reads historical price-change events. In this demo process, reads a local CSV and computes per-category mean/std of price, published to the category-baselines Kafka topic, which the Flink job consumes as a broadcast stream.

LLM Layer

For every flagged anomaly, Claude turns that number into more human readable something like “90% price drop in 3 minutes, looks like a decimal error”.

We have given a special instruction to our AI agent.

In under 60 words, explain in plain English what likely happened (e.g. decimal
error, promotional glitch, competitor scraping issue, genuine repricing,
runaway markup) and recommend one concrete action (e.g. auto-hold listing,
notify seller, no action needed - looks like a legitimate sale). Be direct and
specific, no hedging filler.

Streamlit Dashboard

You see the alerts and the LLM layer analysis in the real-time split by severity(critical/high/medium) with explanation.

How Does it help

We are catching the price errors in the real-time, instead of a batch process. Which helps you control the damage in sub second.

It will also help a human in the loop. You don’t need a manual intervention or someone to look at the each of the prices changes and analyze. LLM layer takes care of it for you.

Also, remember the LLM layer is only been called for the flagged events, and not every price change. By this way the cost stays low.

Architecture

Setup and Execution

Clone the repo.

git clone https://github.com/ajithshetty/price-anomaly-agent.git

1. Copy .env.example to .env and set ANTHROPIC_API_KEY.
2. Start the Kafka Stream(redpanda process)

docker compose up -d redpanda

3. Build and start Flink

 docker compose up -d flink-jobmanager flink-taskmanager
 docker compose exec flink-jobmanager flink run -py /opt/flink-job/anomaly_job.py

Note: The taskmanager only has 2 task slots (taskmanager.numberOfTaskSlots: 2). If you resubmit the job (e.g. after editing anomaly_job.py) without stopping the previous run first, the new flink run will fail with NoResourceAvailableException: Could not acquire the minimum required resources because the old job is still holding both slots.

4. Run the Spark baseline job once (and periodically, e.g. nightly via cron/Airflow)

docker compose run --rm spark spark-submit /opt/spark-training/train_baseline.py

5. Start the LLM explanation service + dashboard:

docker compose up -d --force-recreate app

6. Start the event producer:

docker compose run --rm producer

Open a dashboard: http://localhost:8000

• http://localhost:8501 (Streamlit, started automatically via docker compose up -d streamlit-dashboard)

There you have it.

Using Flink we are reading the price change events from Kafka. And category-baselines(mean, std) per category, refreshed periodically by the Spark batch job (train_baseline.py).

Flink writes the events whose z-score or pct-change crosses threshold, enriched with the features the LLM explainer needs.

LLM layer is reading the flagged events and converting it into a human readable event with the action point.

This exercise is basically a simple example to show how we can easily integrate the AI with the realtime data processing.

courtesy gemini

Reference

[embed]GitHub - ajithshetty/price-anomaly-agent: Real-time detection of weird price changes (glitch… Real-time detection of weird price changes (glitch pricing, scraping errors, seller mistakes) on a marketplace, with an…github.com

[embed]*Streamlit A faster way to build and share data apps* Streamlit is an open-source Python framework for data scientists and AI/ML engineers to deliver interactive data apps …*streamlit.io

[embed]Apache Flink® - Stateful Computations over Data Streams Recent Flink blogs Apache Flink Agents 0.3.1 Release Announcement July 25, 2026 - Wenjin Xie. The Apache Flink…flink.apache.org