Meet Bumblebee: The Multi-Agent AI Architecture That Changed Fraud Detection at Razorpay

Our fraud detection team was facing 20,000 alerts which required 8500 hours reviewing merchant websites manually every month. Then we built an AI that did it in seconds.

Contributors: Sumit Raj, Parin

It was 2 AM when our Head of Risk Operations sent the message that would change everything.

“We’re drowning,” the Slack notification read. “12,000 merchant reviews this month. Each takes 20 or more minutes. Do the math.”

I did. 8500 hours of human attention, every single month, just to check if merchant websites looked sketchy. Privacy policies buried three clicks deep. Social media accounts that didn’t match the business name. Pricing that seemed too good to be true. Our risk team was essentially acting as highly paid web browsers, and we were hiring as fast as we could just to keep up with growth.

The kicker? Different agents would look at the exact same merchant and reach completely different conclusions. One would flag a generic privacy policy as suspicious. Another would shrug it off. There was no consistency, no learning, no way to scale without burning money on headcount.

That’s when we started building what we now call Bumblebee, an AI system that reviews merchants automatically. Today, it handles those 12,000 monthly reviews in seconds, with better accuracy than our human team achieved. But the story of how we got here reveals something unexpected about building AI systems: the technology isn’t the hard part. Architecture is.

The Trap We Almost Fell Into

Here’s what most teams do when they decide to “add AI” to something. They pick a hot framework, wire it up to their data, throw in some prompts, and pray it works. If it doesn’t scale, they blame the model. If it’s inconsistent, they blame the prompts. If it’s expensive, they blame the API costs.

We almost did the same thing.

Our first attempt used n8n, a visual workflow builder that let us prototype fast. Drag some nodes, connect them with arrows, add a few API calls, done. Within weeks, we had something working: webhook comes in, fetch merchant data, review their website, check domain registration, run it through LLM, post the results. Beautiful.

Then we tried to actually use it in production.

The first merchant review failed because their website had a weird character encoding. Fine, add a branch to handle that. The second failed because they didn’t have a privacy policy at all. Add another branch. By week three, our clean 10-node workflow had exploded into 40+ nodes with duplicated logic everywhere. Changing how we handled one specific edge case meant hunting through the entire graph to find every place that logic appeared.

Worse, when something broke, debugging was like surgery with oven mitts. The logs told us “Node execution failed” but reconstructing what actually went wrong required scrolling through thousands of lines of generic error messages. We once spent three days tracking down why HTTP requests were intermittently timing out, only to discover it was a known bug in n8n’s connection pooling that required a workaround nobody had documented.

The n8n prototype taught us something valuable: visual programming feels fast until you need to actually maintain it. We’d validated that AI could do the job, but we needed to rebuild from scratch.

When Smart Agents Become Dumb

Phase two was Python. Proper code, proper libraries, proper logging. We built a ReAct agent, which is a fancy way of saying “an AI that thinks out loud and uses tools.”

The agent would receive a merchant case and reason through it step by step. “I should check their website first. Okay, I found a privacy policy, let me extract it. Now I’ll check domain registration. Hmm, the domain is only 2 weeks old, that’s suspicious. Let me pull fraud metrics…” It was mesmerizing to watch. The AI was genuinely thinking through the problem like a human analyst would.

Then we hit a wall we didn’t see coming: token limits.

See, the agent kept everything in its “memory” as it worked. It would review a merchant website and hold onto the entire HTML source. Then check domain info and add that to memory. Then fraud metrics. Then social media. By the time it was ready to make a final decision, it was trying to process 70,000+ tokens of context.

For merchant websites with lots of content, we’d just… run out of memory. The AI would crash with a context overflow error, or it would silently truncate the context and make decisions based on incomplete information. Neither was acceptable.

The other problem was speed. The agent did everything sequentially. Check the website, wait for the result, think about what to do next, check the domain, wait for the result, think again. Even though checking the website and checking the domain had nothing to do with each other and could have happened simultaneously, the agent’s architecture forced everything into a single-file line.

We’d solved the maintainability problem from Phase 1, but we’d created new ones. Our success rate was only 88%. The average evaluation time was 35 seconds. As we added more data sources to improve accuracy, things got slower and more expensive, not better.

The architecture wasn’t going to scale. Again.

The Breakthrough: When We Stopped Thinking Like Programmers

The insight that changed everything came from watching our human risk team work.

They didn’t have one person doing everything. They had specialists. One person was great at analyzing website content. Another knew all the domain registration red flags. A third could spot fake social media profiles instantly. They’d work on cases in parallel, then come together to compare notes and make a final call.

What if we built the AI system the same way?

Instead of one god-agent trying to do everything, we built multiple specialist agents:

The Planner decides what information we need to gather. It looks at the merchant case and creates an execution plan: check these data sources, with these priorities, using these timeouts.

The Fetchers work in parallel, each owning one data source. The Website Fetcher knows everything about scraping and parsing web content. The Domain Fetcher understands WHOIS data. The Fraud Database Fetcher knows how to query our internal systems. Each one is excellent at its specific job.

Here’s the critical insight: the Fetchers don’t send raw data. Instead of dumping entire HTML data into the system, the Website Fetcher extracts just the relevant bits. Privacy policy? Extract that section. Contact info? Grab it. Product pricing? Got it. Then it packages everything into a tiny, structured JSON payload.

This pruning pattern solved our token limit problem entirely. Instead of accumulating massive amounts of raw data, we maintained small, dense summaries.

The Analyzer receives these compact summaries from all the Fetchers and makes the final call. It runs simple rules first, then uses the AI for nuanced judgment. Because the data is already cleaned and structured, the AI works with minimal context.

The results were dramatic:

• Token usage dropped 60%
• Average evaluation time: 8–12 seconds (down from 35)
• Success rate: 99%+ (up from 88%)
• We could now process thousands of evaluations concurrently

But the real win wasn’t the metrics. It was that adding new capabilities became trivial. Want to check merchant SSL certificates? Write a new Fetcher. Want to analyze page load times? Another Fetcher. The Planner automatically incorporates new tools, and the Analyzer adapts without changes.

The architecture finally scaled.

What This Actually Means for Building AI Systems

Here’s what we learned through three architectural iterations and countless failures.

Lesson 1: Start simple, rebuild deliberately

N8n was the right choice for our prototype even though we knew it wouldn’t scale. Proving the concept mattered more than perfect architecture. But we had the discipline to recognize when we’d outgrown it and rebuild from scratch rather than patch over fundamental limitations.

Lesson 2: Token limits are architectural constraints, not implementation details

Everyone focuses on which model to use and how to write better prompts. Almost nobody talks about token budgets. In production systems with messy real-world data, token limits are where architectures break. Design for token efficiency from day one: prune early, prune often, never pass raw unstructured data to LLMs.

Lesson 3: Specialization beats generalization at scale

A single agent trying to do everything will hit walls you can’t solve with better prompts or bigger models. Focused agents with clear responsibilities produce systems that are faster, cheaper, and more reliable.

Lesson 4: Parallelism matters more than model intelligence

Running multiple small agents in parallel often beats running one large agent sequentially, both in latency and cost. Stop trying to find the perfect all-knowing model. Build systems where simpler models work together.

Lesson 5: Observability is mandatory

Without structured logging and the ability to replay decision sequences, debugging production AI is impossible. Every agent logs what it did, why it did it, and what it found. When something goes wrong, we can replay the exact sequence of events and understand what happened.

The Part Nobody Talks About

Building Bumblebee wasn’t a smooth path from vision to execution. We threw away two complete implementations. We spent weeks debugging problems that turned out to be fundamental architectural issues, not bugs we could fix. We had heated debates about whether specialized agents were over engineering when a single smart agent “should” work.

The biggest lesson? The first architecture that works is rarely the architecture that scales. And that’s okay. What matters is having the discipline to recognize when you’ve hit limits and the courage to rebuild rather than patch.

Today, Bumblebee reviews merchants in seconds instead of minutes. It catches fraud patterns our human team would miss. It works 24/7 without breaks. And it gets better every day as we add new Fetchers and improve existing ones.

But the real story isn’t that we built a cool AI system. It’s that we learned to recognize when architecture was the problem, not the technology. That lesson applies to every production AI system, whether you’re detecting fraud, generating content, or processing documents.

The technology is ready. The models are capable. The frameworks exist. The hard part is designing systems that survive contact with reality.

That’s what Bumblebee taught us. That’s what three failed architectures bought us.

And honestly? That’s worth way more than getting it right the first time.

Ankur (left) and Sumit (right) with Project Bumblebee taking the center stage at Razorpay office. P.S — We missed you Parin!

Bumblebee processes 12,000 merchant reviews monthly at Razorpay, reducing fraud detection time from hours to seconds while improving accuracy. The multi-agent architecture handles thousands of concurrent evaluations and gets smarter with every case it processes.

If you’re building production AI systems and hitting similar scaling challenges, I’d love to hear about your architecture journey. What broke first? What surprised you? Drop a comment below.

Editor: Parth Sawhney