Early learnings: can an AI Moderator capture human insight?

I recently shared a peek into one of my latest experiments: using an AI moderator to run asynchronous user interviews. This concept is potentially ambitious: Can AI-moderated interviews scale our research speed without sacrificing human-to-human richness?

I created the AI moderator in HeyMarvin by feeding it a detailed discussion guide, and then launched the experiment. I reached out to a group of our customers and invited a few people to try this new research method, being transparent about it being AI. The promise of increased speed, scale, and efficiency was thrilling to us internally. But what did our users, the sharp-minded healthcare professionals we design for every day, actually feel about talking to a bot?

Well, to be candid, we only got a handful of users to test it out. The good news is they were all willing and enthusiastic participants! As with any new tool that attempts to bridge the gap between people and process, the response was a beautiful, messy collection of validation and constructive criticism.

EarlAI feedback (I had to…)

On one hand, the responses clearly validated what we hoped for from a customer perspective: less scheduling friction and more flexibility. Making participation easier was the goal, and on that front, we saw a clear win. In fact, one of our users was really excited about this new way of providing feedback to our team. They messaged their Client Success Manager to let them know how much “fun this encounter was, [and that they] think this is an amazing way to gather data.” The participant highlighted the ease of conversation, noting they “felt the bot was easy to talk with and felt like it was capturing what [they were] saying.” The participant encouraged us to use the AI moderator more! This was gold. When the experience is “fun” or simply fresh, it can help bring more voices into the room. This feedback validated that the tool is a pathway to a more accessible and engaging research method.

The critical lens of a power user

Then came the feedback that keeps us grounded. Another participant, a power user for AI tools, offered a two-sided breakdown. They saw the advantages immediately: “I could complete the interview at any time, it saves time for you or your staff, the format is adaptive based on whatever topics I bring up…and it’s possibly a good way to identify several individuals for a more detailed follow up.” This confirms the efficiency gain, the accelerating power we were aiming for. But they also cut straight to the trade-off; they called it “a pretty poor conversationalist.” You see, the AI moderator follows a pattern:

1. Reiterate what the participant said

a. Ex: “Got it. So having the ability to compare apps side-by-side would make your shopping experience easier.”

2. Ask a follow-up question (this can come in the form of a probing question if the participant didn’t fully answer the original question, or the next question in it’s queue)

a. So, tell me, what sort of information would you want to compare app-to-app?

It’s not that the AI responds in an incorrect way, it’s the repetitive pattern that the participant recognized and objected to. It doesn’t feel very natural, and potentially makes the participant feel as if they aren’t being heard (the opposite of what the pattern is intended to do).

We still need more feedback, but for now, the conclusion is…

AI-moderated interviews are not a replacement for human-to-human research, but they are an excellent accelerant and filter. They allow us to cast a wide net for early insights and better understand which participants are the strongest fit for deeper follow-up. The early insights help steer design direction and product roadmaps, helping us get to an MVP faster.

And that, for now, is exactly how I plan to keep using this tool. I’m exploring along with the technology, and I invite you to keep following along as we remove unnecessary hurdles for busy clinicians to share their feedback with us.