After 11 years doing Adobe Experience Cloud work manually, I built the agent that does the parts that don’t need me.

Introducing, Rovr, an unofficial Adobe Experience League chatbot.

I spent over 11 plus years working on perfecting Adobe Experience Cloud implementations — AEP, AJO, CJA, Analytics, Targe. Over the course of this long stretch, my go to resource for any problem has been Adobe Experence League. But, every engagement had the same shape: whenever stuck with a question, do a slow grind through ExL documentation, configuration, and trial and error to solve it.

At some point, it dawned on me what if there was an AI agent to simplify searching across multiple docs and streamline the process? What if there is an agent that can take on the parts of that grind that don’t actually need a human — the “what does this parameter do,” “how do I structure this schema,” “why is this event not firing” layer of the work. Not replace the judgment. Just remove the friction.

That question turned into Rovr, an unofficial assistant for Adobe Experience Cloud practitioners. I, honestly, wasn’t expecting a lot when I launched the tool and made an announcement on LinkedIn.

But the feedback I received painted a different picture. Practitioners in the Adobe Experience Cloud space have, just like me, been grappling with the complexity of ExL documentation and were actually hoping for an easier way.

Practitioners who have been practicing and are experts in the space of Adobe Experience Cloud reached out stating how my tool has solved a long standing pain point

And not just practitioners, the ExL Community itself gave a thumbs up to the idea

These types of encouragement assured me that I am on the right path. And that led me to seriously start dedicating time and effort on the project.

Five weeks after launch, I have enough production data to say something more interesting than “it works”: I have data on what happens when real people use it, what it costs to run properly, and what I got wrong.

The architecture decision that actually mattered

The easy way to build a RAG chatbot is to route every query through the most capable model you have and worry about cost later. I didn’t want to build that, partly because “later” is where most side projects die, and partly because cost discipline is the actual engineering problem — the demo is easy, the sustainable version is hard.

So the system routes by task. Simple classification and retrieval-adjacent work — figuring out what a query is actually asking, matching it to the right domain (AEP vs AJO vs CJA vs Analytics) — goes to Haiku. Generation, where the answer needs real synthesis across retrieved context, goes to Sonnet. Over the first month: 314 queries went through Sonnet, 199 through Haiku, out of 672 total.

The result: $17.34 in total LLM spend across those 672 questions. Roughly 2.6 cents per query.

That number isn’t just a flex line — it’s the difference between a system that could plausibly run for a community of practitioners indefinitely, and one that becomes an accounting problem the moment usage grows. Anyone can wire up an LLM and a vector store. The unglamorous part — routing by complexity, catching a citation validator that was killing valid URLs on timeouts, fixing a hallucination pattern with a second-pass groundedness check — is what makes the difference between a POC and something people can actually rely on.

What happened when people actually used it

Here’s where it stopped being predictable.

Launch week brought 28 new users and 165 questions across 24 distinct people. Good reach, exactly what you’d hope for out of a cold launch. The following week, usage dropped to 52 questions — a real dip, not a rounding artifact of the data. I don’t have a fully confirmed cause, but the shape (a spike of curiosity-driven trial, followed by a falloff before the people who found it useful settle into a rhythm) is a pattern I’d expect from any tool without a habit loop yet.

It recovered. By week five, weekly volume peaked at 208 questions — the highest of the five weeks. But that peak didn’t come from a bigger crowd. It came from just 8 users. The average question count per user across the whole period was 10.9 — meaning that peak week alone, a handful of people were each running through more than 25 questions.

Reach and depth turned out to be two different curves that don’t move together. Launch week told me people were curious. Week five told me a small group had actually made this part of how they work. Those are different signals, and conflating them (as “672 questions answered!” style stats naturally invite you to do) would have hidden the more useful insight: 56% of users returned for a second question, and 24% became repeat power users. That’s the number that actually matters for whether this is a tool or a novelty.

Feedback ran 69% positive (67 up, 30 down) — not a perfect score, but a real one, from real usage rather than a curated demo audience.

The topic breakdown added a layer I didn’t expect to find useful. Experience Platform fundamentals drew 167 questions; CJA drew 21 — roughly an 8x gap. That’s not necessarily a statement about Rovr’s coverage. It’s more likely a signal about where the community’s actual knowledge gaps sit right now: more people are still getting oriented on AEP basics than are deep enough into the stack to be troubleshooting CJA specifics. If you’re earlier in this stack yourself, that ratio is a decent hint about where to spend your own learning time first.

What’s still broken

The number that bothers me most isn’t a bad number, it’s an incomplete one: 15 of the 62 people who signed up never asked a single question. They cleared onboarding and then didn’t come back to try it.

That’s a different problem from acquisition, and I think it’s one a lot of early-stage tools — mine very much included — underweight. Getting someone to register is not the same as getting them to their first real question. I don’t have a fix shipped yet. My current hypothesis is that the gap is a first-touch problem — no obvious “type here” moment, no sample prompt nudging someone past the blank box — but I’d rather test that than assume it.

There’s a related, smaller version of the same problem: about 19% of users asked exactly one question and never returned. Some of that is probably genuine one-off curiosity that was satisfied in a single answer. Some of it is probably a bad first answer. I can’t fully separate those two populations from the data I have, which is itself a useful thing to know — it means my next build priority isn’t a feature, it’s better instrumentation on that specific moment.

Why this connects back to the work I actually do

None of this — the routing logic, the cost curve, the retention numbers — exists separately from the 11 years before it. The reason I could build the topical classification correctly, catch the hallucination pattern, and know which questions actually matter to a practitioner is that I spent over a decade being the person answering those questions manually, on real client engagements.

The thing I keep coming back to is that building the agent version of that expertise turned out to be a different skill from having the expertise itself — and a more interesting one. Knowing that a client’s XDM schema is misconfigured is domain knowledge. Building a system that can reliably surface that same insight to a stranger, at 2.6 cents a query, without hallucinating an answer when it’s uncertain — that’s a different kind of craft, and it’s the one I’m more curious about right now.

Rovr is still small. 62 users, five weeks, $17 spent. But the shape of the data — the dip, the depth-over-reach pattern, the activation gap I haven’t solved — feels like a more honest account of what building something real looks like than a clean growth chart would have been.

Rovr is a personal project. If you work in the Adobe Experience Cloud ecosystem and want to try it, or if you’ve hit similar activation-gap problems in your own tools, I’d like to hear about it.