Prompt created with Visual Prompt Architect. Run in Midjourney.

How to Know If Your AI Product Is Working: The 4 Metrics That Matter

Acquisition, conversion, retention & revenue. How to read your analytics to decide whether to celebrate or start over.

Every solo builder has opinions about their product. We debate copy, tweak layouts, simplify onboarding, and move buttons around, hoping we’re making things better.

But after you hit publish, there are only two possibilities: either users behave differently, or they don’t.

Over the past few months, I’ve been building Universal Prompt Designer, an AI-powered product that helps people create better prompts for tools like ChatGPT, Claude, Gemini, and Figma Make.

The first version wasn’t perfect. I set myself a deadline, launched it in March, and put something real in front of users because I knew I’d learn more from real behavior than another month of polishing.

It also raised plenty of questions.

• Did visitors understand the value proposition quickly enough?
• Was the marketing page explaining too much or too little?
• Were people actually finding the product useful, or were they just experimenting once and leaving?

In June, I shipped a significant update that included a redesigned marketing page, a simpler onboarding experience, and a migration to a new chat platform with a better user experience.

“I want data to show me the way.”

Rather than relying on intuition, I looked for evidence that user behavior had changed in the ways I expected. Not just more visitors, but better conversion. Not just more signups, but deeper engagement. Not just curiosity, but retention.

A month later, those metrics gave me the confidence to answer one of the biggest questions I’d had since launching the product: Could Universal Prompt Designer generate enough revenue to sustain itself?

By July, I had my answer.

Every Product Change Is a Bet You Should Track

Every product change starts with a hypothesis; a prediction about how users will behave. Analytics tell me whether that prediction was right. Sometimes it tells me I was wrong, and that’s the point. I’d rather the numbers correct me early than let a bad assumption persist.

What I’m looking to learn from analytics is simple: Did the behavior I wanted to influence actually change?

For example, when I simplify messaging, I expect more visitors to understand the value proposition. When I reduce friction during onboarding, I expect more people to use the product. If I solve the right problems, I expect people to come back on their own.

That turns metrics from numbers on a dashboard into evidence that either supports or challenges the decisions I made.

That mindset keeps me focused on outcomes instead of activity. It’s easy to celebrate more traffic, more signups, or more page views. Those numbers matter, but only if they represent a meaningful change in user behavior.

For this update, I was trying to answer four questions:

1. Are people finding the product? (Acquisition aka “New Users” in GA4)
2. Do they understand enough to create an account? (Conversion Rate)
3. Do they come back after their first visit? (Retention)
4. Do enough people find value to pay? (Revenue)

Together, those metrics told me whether the product was moving in the right direction.

First, Are the Right People Even Showing Up?

The first question I wanted to answer was whether I was reaching the right audience.

Between May and June, Universal Prompt Designer grew from 502 new users to 1,705, a 294% increase over the previous month.

Reddit drove the largest share of that growth. One discussion during the FIFA Club World Cup was an especially good fit for Universal Prompt Designer’s value proposition, and over two days it sent roughly 260 visitors to the site.

What interested me more, though, was what happened outside of that spike. Direct traffic increased substantially, suggesting people were returning intentionally rather than discovering the product for the first time.

I also started seeing visitors arrive through X, Bitly links, and, perhaps most encouragingly, Google and Bing. The search traffic is still small, but it suggested the site is beginning to rank organically.

What changed

The Reddit spike wasn’t just a traffic spike. Combined with growing traffic from direct visits, search, and social, it gave me confidence that the product’s value proposition was resonating with the audience I was trying to reach.

That shifted my attention from who was finding the product to what happened after they arrived.

Showing Up Is One Thing. Getting It Is Another.

My hypothesis was that the marketing page was over-explaining the product.

Visitors didn’t need more information. They needed the right information, presented sooner.

When I launched the MVP in March, I was trying to answer every question a visitor might have because I didn’t yet know which questions actually mattered. After several months of watching how people used the product, reading feedback, and refining the positioning, I realized I could do what I usually recommend in UX work: remove the unnecessary and let the important information stand out.

I rewrote the page to introduce the value proposition much earlier, speak directly to the audience I was building for, simplify the use cases, remove sections that weren’t contributing to the decision-making process, and rely more on visuals than paragraphs to explain how the product worked. I’ll break down those UX decisions in detail in a separate article. For this article, I only cared about one question:

Did those changes influence user behavior?

The answer was yes.

Traffic increased dramatically, but the more important result was that the conversion rate also more than doubled. Because both metrics improved together, I could be reasonably confident the improvement wasn’t simply driven by higher traffic. More of the people arriving were deciding to create an account.

What changed

Once I understood who I was building for and the problem they were trying to solve, I stopped trying to anticipate every possible question. Instead, I focused on helping visitors quickly recognize that the product was built for them.

The redesign reinforced something I already suspected: good design builds trust.

A thoughtful visual design, clear information hierarchy, and consistent brand language signal that a product has been intentionally designed and is worth taking seriously. Those elements don’t replace a strong value proposition, but they make it easier to believe.

Creating a free account, though, is still a relatively small commitment.

OK, But Do They Actually Come Back?

If I had to choose a single metric that tells me whether a product is becoming useful, returning users would be it.

Anyone can sign up for a product. Anyone can spend ten minutes clicking around. Coming back on a different day is a different decision, and that’s exactly how I chose to measure retention.

It means the product solved a problem well enough that someone remembered it, returned to it, and decided it was worth using again. That’s the beginning of a product becoming part of someone’s workflow, which was exactly what I intended Universal Prompt Designer to become.

When I relaunched Universal Prompt Designer in June, the work went beyond redesigning the marketing page. I also migrated the product from LaunchLemonade to Pickaxe.

LaunchLemonade had been the fastest way to get my MVP into users’ hands, which was exactly what I needed at the time. My goal wasn’t to build the perfect product. It was to start learning from real users.

Over the next few months, though, it became clear that the platform wasn’t the right long-term fit. The onboarding introduced unnecessary friction before users experienced any value, and the chat experience wasn’t where I wanted it to be. At the same time, I was steadily strengthening the knowledge base and system instructions behind Universal Prompt Designer.

Rather than continue optimizing around those constraints, I migrated to Pickaxe and kept iterating.

The difference showed up almost immediately.

Between March and May, 120 people created an account on LaunchLemonade. None of them returned on another day.

That zero was the clearest signal I got the whole project. It was the data telling me to start over.

By June, after migrating to Pickaxe and continuing to improve the product itself, 10% of users returned on a different day to use Universal Prompt Designer again.

What changed

No single change explains the improvement. Stronger system instructions, a better chat experience, and dozens of small refinements all contributed.

The analytics showed me those changes were having the intended effect. People weren’t just creating accounts. They were remembering the product, coming back to it, and beginning to make it part of their workflow.

That changed how I measured success. Signups told me people were interested. Returning users told me the product had become useful enough to rely on.

The Moment of Truth: Will They Pay?

Before someone decides to pay, they have to discover the product, understand what it does, find enough value to keep using it, and eventually decide it’s worth coming back to.

By the time someone purchases, they’ve already answered dozens of small questions about whether your product deserves a place in their workflow.

By July, Universal Prompt Designer reached one of the first milestones I set for the project: it was generating enough revenue to cover its own infrastructure costs.

What this told me

Universal Prompt Designer is no longer just an experiment. It’s something people valued enough to pay for.

For me, it meant the product had crossed an important threshold: it was creating enough value to sustain itself and fund its own continued development.

Revenue is the outcome of everything that comes before it.

No single feature or redesign explains why someone subscribed. The stronger system instructions, a better chat experience, clearer messaging, better presentation, and months of incremental improvements all compounded over time.

Revenue told me that those decisions were working together to create something people were willing to pay for.

Let the Data Show You the Way

If you’re building a product today, I’d encourage you to start collecting that data evidence as early as possible.

Install Google Analytics. Create a dashboard that surfaces the handful of metrics that actually matter for your business. You don’t need to become an analytics expert overnight, but you do need a way to measure whether your assumptions hold up once real people start using what you’ve built.

Without that feedback loop, every product decision is based largely on intuition. Experience and instinct are what help us form hypotheses. Data tells us whether those hypotheses were right.

That’s become one of the principles behind how I build products.

About My Work

I’m a product designer focused on AI, UX, and product strategy. I build AI products from idea to paying customer, documenting what works, what flops, and what the data teaches me along the way.

If you’d like to see the redesigned product behind this article, you can try it here → **Universal Prompt Designer**

Create and monetize custom AI chatbots with** → Pickaxe **(Affiliate link)