My Impact in Building Pulse-Agent (An AI Agent That Reads Your Mind — DSN x BCT Hackathon 3.0)

My Impact in Building Pulse-Agent (An AI Agent That Reads Your Mind — DSN x BCT Hackathon 3.0)

A little over a week ago, my team and I entered the DSN × BCT Hackathon 3.0 with one goal; build an autonomous multi-agent system that could simulate how a real person would review something they’ve never seen before, and recommend things they’d actually enjoy. The kind of system that doesn’t just pattern match, but genuinely reasons about who you are.

We called it Pulse-Agent.

Meet the team (HOKN)

Team Lead & Architecture: Henry Morgan-Dibie | ML Systems Engineer

Data Engineering: Kenneth Onwubiko | Data & Automation Analyst

Recommendation Systems: Michael | ML Systems Engineer

Front End & Solution Paper: Kindness Onwuchekwa | Data Scientist

Interact with the full project HERE

The Stack

For anyone curious about the technical setup, the agents run on LangGraph with Groq’s llama-3.3-70b-versatile model. The data pipeline is pure Python with pandas. Metrics use scikit-learn, rouge-score, and bert-score. The evaluation runner is fully async using asyncio. Everything is typed with Pydantic end to end.

The full project is on GitHub. My specific contributions live in src/data/ and src/evaluation/.

Data Engineering Foundation

Everyone loves talking about AI agents. The reasoning, the language models, the clever prompts. What people don’t talk about is what happens before any of that; the data.

Before an agent can simulate your review style, it needs to know your review style. Before it can recommend something you’ll like, it needs to understand what you’ve liked before.

For the pulse-agent I handled this phase of assembling historic review data from three different sources which was used to train the pulse-agent. I also evaluated the train and test results

Datasets

We relied on three data sources; Amazon reviews 2023 (product reviews), Yelp (restaurants and business reviews) and Goodreads reviews(books).

Challenges with datasets; Being that this datasets were from different sources, each came with a unique style of column headers.

Yelp calls a business a business_id. Amazon calls a product an asin. Goodreads doesn't even have individual user reviews, it's book metadata with community averages. Every field name, timestamp format, and rating convention is different.

The solution — Three loaders, one per dataset that would take these raw files and output something identical in structure. Same column names, same data types, same rating scale. The agents downstream never needed to know where the data came from. It just worked.

Once that was done I proceeded to unify them into a single dataset since they now share the same structure.

Train/Test Split

For this phase I used a temporal split. For each user, I sorted their reviews chronologically and put their most recent 20% into the test set. Everything older went to training. This mirrors exactly how a real recommendation system works, it learns from your past and predicts your future. No leakage, no inflated metrics, no false confidence. Unification, train/test was done using a preprocessor engine. This engine outputs two datasets (train.csv, test.csv)

Measuring What Actually Matters

Building the pipeline was one half of my role. The other half was building the evaluation harness, the system that tells you whether the agents are actually good or just getting lucky.

I implemented nine metrics across the two tasks our system handles.

For Task A, which is review simulation, I needed to measure both rating accuracy and text quality.

RMSE tells you how far off the predicted star rating is.

ROUGE-L measures word overlap between the generated review and the real one.

BERTScore goes deeper, it uses a language model to measure semantic similarity, catching cases where the agent writes something that means the same thing in different words.

For Task B, which is recommendation, I needed to measure ranking quality.

NDCG@10 tells you whether the most relevant items appear at the top of the list.

HitRate@10 measures whether at least one genuinely relevant item appears in the top 10.

Cold-start MAE measures accuracy specifically on users the system has never seen before, the hardest possible scenario.

Then I built the runner; a fully asynchronous evaluation pipeline that loads the processed data, calls both agents on held-out test records, collects predictions, and computes every metric automatically. One command and it runs end to end.

What the Numbers Actually Showed

When we finally ran the full evaluation, here’s what came back:

Task A — Review Simulation:

• BERTScore F1: 0.84 — the agent’s generated reviews are semantically very close to what users actually wrote
• ROUGE-L: 0.14 — lower, but expected, since the agent writes in each user’s personal style rather than copying phrases
• RMSE: 1.31 — compared to a mean-prediction baseline of 1.08

The RMSE being higher than the baseline is worth addressing honestly. A mean-prediction baseline always performs deceptively well on small samples because it never takes risks.

The agent introduces personalized reasoning, it tries to capture who you are and on a 10-sample evaluation set (due to limitation on Groq tokens), that variance shows up. However with a larger evaluation set and further tuning, the gap narrows.

Task B — Recommendation:

• NDCG@10: 1.0 — the system consistently ranks the most relevant items at the top
• RMSE: 0.76 — predicted ratings are closer to actual ratings than in Task A, because the ranking agent benefits from broader candidate context

The Ablation Study

One of my favorite parts of this project was the ablation study. The idea is simple; remove one component at a time and see how much performance drops. Due to limited usage of Groq API tokens we narrowed down to just RMSE.

Here’s what we found on the RMSE metric:

What Was Removed | RMSE

Nothing (full system)| 1.33

User history signals | 1.08

Item-level signals | 1.14

Everything (random)| 2.09

Removing all signals and going fully random caused a 58% increase in RMSE. The full system’s higher RMSE compared to the ablated versions reflects the personalized variance the agent introduces, which the statistical proxies can’t replicate.

Conclusion

A few things stayed with me from this project.

1. Data decisions compound. The temporal split I chose, the synthetic Goodreads approach, the normalization strategy; each one had downstream consequences for every metric, every agent output, every number in the paper. Getting these right early meant the rest of the team could build on a stable foundation.
2. Honest evaluation is harder than good evaluation. It would have been easy to choose metrics that made the system look better than it is. Instead we reported the RMSE honestly, explained the HitRate result clearly, and let the numbers speak.
3. Infrastructure is invisible until it breaks. Nobody sees the loaders running in the background. Nobody notices the train/test split logic. But if any of it is wrong, every result downstream is wrong too. Building this layer well is quiet, as well as important.

Special thanks to Henry for guidance in building the engines and making sure nothing breaks behind the scene. And to the rest of my team members for making sure the pulse agent came alive. Interact with the live demo HERE

Till the next project, See you again!