Accelerating insights: My dive into AI-moderated user interviews

The most critical question we ask as designers — Does this actually work for the person using it? — is often the hardest to answer quickly and at scale. User research is how designers validate assumptions and ensure we’re building the right thing, but its traditional process often bottlenecks our discovery phase with scheduling logistics and analysis overhead. To accelerate design/product decisions without compromising on research depth, I’ve been looking at how we can thoughtfully scale our user testing efforts. My current experiment involves the use of an AI-moderated interviewer from HeyMarvin.

What is the AI-moderated interview tool?

This tool is designed to handle the conversational heavy lifting by conducting a natural, adaptive interview with users. This is less about automating research and more about automating the administrative burden of research. I believe speaking to users directly should always be a part of the mix, but if we could reach more users faster with the addition of an AI interviewer, why not try it? Instead of juggling calendars and spending hours facilitating sessions, I can now scale the number of conversations happening in the discovery phase. This lets me dedicate my time to reviewing insights from the AI tool and ensuring our final design decisions are deeply informed by the qualitative data.

The How: Structuring an AI conversation

Using an AI moderator requires a solid discussion guide that the AI tool will use to run the conversation (it’s like an instruction manual or recipe for the AI moderator). This starts with research goals and open questions. Think about what you want to learn from users and why; how are you going to use the answers in your design process? Once you have the research goals nailed down, you can use AI to create a discussion guide with your goals as inputs.

Sample research goals and open questions from my study assessing the user experience of a comparison tool for third-party healthcare apps that integrate with an EHR (electronic health record system, athenaOne).

1. Evaluate usability of the comparison tool

a. Can users easily add, remove, and view apps side by side? b. Do they understand the layout, labels, and comparison categories?

2. Assess decision-making support

a. Does the tool surface the right features, capabilities, and metrics to help users narrow options? b. Are users confident making a shortlist or choice after using it?

3. Test clarity of information hierarchy

a. Are the most important details (integration depth, pricing model, user counts, etc.) clear and discoverable? b. Do visuals (grids, icons, media) aid or distract from comprehension?

4. Gauge trust and perceived value

a. Do users trust the accuracy and completeness of what’s being compared?b. Does it feel like an objective, helpful tool versus marketing fluff?

5. Identify unmet needs or feature gaps

a. What additional filters, metrics, or guidance would make the tool more useful? b. Would users want AI assistance (e.g., a bot to explain differences or answer questions) alongside the grid?

Here’s an example of the discussion guide I generated with ChatGPT (with the research goals as an input):

📋 Discussion Guide — Marketplace 3.0 Comparison Tool

Study Link: [Figma Make Prototype]

Format: Moderated user research session

Audience: Marketplace customers (mid-size healthcare practices, decision-makers, staff evaluating apps)

1. Introduction (5 min)

• Welcome and thank participant for their time.
• Briefly explain purpose: *“We’re testing a new app comparison tool for the Marketplace. We want your honest feedback on how usable, useful, and trustworthy it feels.”
• Emphasize: we’re testing the tool, not the participant. -Remind them that the session is being recorded. -Remind them to open the prototype link they were sent in advance of this session. The prototype will need to be accessed throughout the session.

2. Warm-Up Questions (5 min)

• Tell me a little about your role and how you’re involved in evaluating or selecting new tools for your practice. -When was the last time you looked for a new app or solution? How did you make that decision? -Have you ever compared apps side by side before? If so, how?

3. Task Scenarios (25–30 min) — Have the user open the prototype

Task 1: Adding & Viewing Apps

• Please add [X apps] to the comparison grid.
• What do you notice about the process? Easy/difficult?
• Probe: Was it clear how to add or remove apps?

Task 2: Exploring the Grid

• Now that you have apps side by side, walk me through what you see.
• What stands out first?
• Are the categories (features, integration, pricing, etc.) clear to you?
• Probe: What feels useful vs. overwhelming?

Task 3: Decision-Making Support

• Based on what you see here, which app would you lean toward and why?
• Do you feel confident making a shortlist or choice from this comparison?
• Probe: What additional info would you need to be more confident?

Task 4: Trust & Comprehension

• How much do you trust the accuracy of what’s shown here?
• Does this feel like an objective comparison or more like marketing content
• Probe: Are visuals (icons, grids, images) helping or distracting?

Task 5: Exploring Beyond Basics

• Imagine you want to go deeper on one app — what would you expect to see
• Would an AI bot that answers questions or clarifies details be helpful? Why or why not?

4. Reflection & Wrap-Up (10 min)

• What was the most helpful part of the tool for you?
• What was confusing, frustrating, or missing?
• If athenahealth could add one new thing to make this tool more valuable, what would it be?
• Any final thoughts or suggestions?

Moderator Tips

• Keep participants talking — use follow-ups like “What makes you say that?” or “Can you show me where you’d expect that?”
• Pay attention to body language/frustration cues.
• Capture direct quotes that might be good for a “voice of the customer” section of a readout.

You have flexibility on how you coach AI

Remember that you can control how the interviewer operates. You can design your own discussion guide in any format, test it, and tweak it if the AI moderator isn’t performing as you’d like. Some examples of what I’ve asked ChatGPT it work into the discussion guide are:

• Telling it to timebox the interview
• Telling it to stick to the script or to allow users drift off-topic (allowing users to drift might be helpful if your research is more exploratory)
• Giving it a more detailed persona of the users being interviewed so it understands who it’s talking to
• Giving it any necessary guardrails, or ask it to alert users of any needed compliance warnings (particularly useful for corporate research)
• Telling it to probe on certain topics
• Telling it to remind the user to use a prototype link you’ve sent them in advance (Note that as of fall 2025, the tool won’t let the participant share their screen.)

So, what does the AI moderator setup actually look like in Marvin?

General info configuration screen

When you create your moderated interview, you have some settings to configure. This is where you add your discussion guide and research goals. You can also choose which AI voice you want to user. :) You’ll be able to test the AI-moderator before you send the link out to users.

Here’s an example of what a participant would see when they go to take the AI-moderated session.

Here’s an example of a session:

[embed]

The Why: Getting to ‘Aha!’ Faster

Why bother with an AI moderator? For me, it comes down to scale and speed.

1. Eliminating the scheduling dance: Instead of spending hours coordinating calendars, I just send out the link and users complete the study when it works for them. This results in faster and bigger reach without the operational inconvenience. This newfound velocity is a massive benefit, especially in healthcare, where our users are busy, time-constrained clinicians.

2. Data aggregation and analytics: This is the real game-changer. HeyMarvin provides immediate, machine-driven analytics, identifying patterns, common phrases, and key themes across dozens of interviews. This shaves down a lot of the manual synthesis work we usually do after a round of research.

Finding the balance: my closing thoughts

The biggest temptation with powerful new tools is to use them for everything. AI-moderated sessions are superb for picking someone’s brain on a specific topic or project; they excel at understanding workflow steps and collecting broad qualitative data at scale. However, they shouldn’t fully replace face-to-face (or video-to-video) user testing. For me, the human-led interview is still the moment of true connection between the designer and the end user, where rapport is built and the best insights are uncovered. The AI session is the wide net, an efficient tool to validate initial assumptions and gather directional feedback at a pace we simply couldn’t achieve before.

Integrating AI into our research process is not about automation for its own sake. It’s about leveraging technology to handle the high-volume, repetitive work, freeing up our time as designers and researchers to focus on the truly complex, insightful, and human part of the job. That, after all, is the part the AI can’t, and shouldn’t, touch.