We Found a 21% Lift in Card Acquisitions — Here’s How We Knew It Was Real

A/B Testing for Card Member Acquisition: A Practical, No-Nonsense Guide

How to design, run, and read an A/B test when your funnel has four steps and your business cares about one number: how many visitors turn into card members.

If you work in growth or marketing analytics at a financial company, you’ve probably run an ad A/B test before. Two ad creatives, one landing page, and a scoreboard everyone checks a little too often. The tricky part isn’t running the test — it’s making sure you’re measuring the right thing, and reading the results correctly.

This article walks through a simple, repeatable framework for A/B testing two ads for card member acquisition, using one clear north-star metric and a funnel that most fintech and card companies will recognize.

*(Note: all numbers in this article are illustrative example data, not real performance figures from any company.)**

Step 1: Map the funnel first

Before you touch a single ad, write down the funnel. For a card acquisition flow, it usually looks like this:

Visitors → Started Application → Submitted Application → Acquisition

1. Visitors: people who land on the page after clicking the ad
2. Started Application: visitors who click “Apply” and begin filling out the form
3. Submitted Application: applicants who complete and submit the form
4. Acquisition: applicants who are approved and become card members

Each step is a filter. Some people drop off between every stage — because of friction, hesitation, credit eligibility, or simply losing interest. Understanding where people drop off is just as important as knowing the final number.

In this example, Ad B keeps more people at every stage of the funnel — not just at the top.

Step 2: Pick one metric to decide the winner

It’s tempting to look at five metrics — click-through rate, cost per click, start rate, submit rate, approval rate — and pick whichever one makes your favourite ad look best. Resist that.

Define your primary metric before you launch the test, and let it be the single source of truth:

Acquisition Rate = Acquisitions ÷ Visitors

Why this metric, and not something like “submit rate” or “approval rate” alone?

• It captures the entire funnel, from ad click to card member, in one number.

• It can’t be gamed by an ad that drives a lot of cheap clicks but few actual card members.

• It directly reflects the business outcome you care about.

You can absolutely track the intermediate steps (start rate, submit rate) as secondary metrics to diagnose why one ad wins. But the acquisition rate is what decides the test.

Step 3: Set up the test properly

A few basics that are easy to skip and expensive to regret:

1. Randomize at the visitor level. Each visitor should be randomly assigned to see Ad A or Ad B, with assignment sticking for their whole session (and ideally across sessions, using a cookie or user ID).

2. Split traffic evenly, typically 50/50, unless you have a specific reason to do otherwise (e.g., a staged rollout).

3. Decide your sample size before you start.Estimate how many visitors you need to detect a meaningful difference in acquisition rate, based on your current baseline rate and the smallest lift you’d care to detect. Committing to this upfront stops you from stopping the test early just because it “looks good.”

4. Run for full business cycles. A week that includes both weekdays and weekends, and ideally more than one pay-cycle or promotional cycle, so you’re not fooled by short-term noise.

5. Keep everything else constant. Same landing page, same targeting, same budget pacing — the ad creative should be the only thing that differs.

Step 4: Read the results the right way

Once the test has run long enough, compare acquisition rates between the two ads — and don’t just look at the headline numbers. Look at whether the difference is statistically meaningful, not just numerically different.

A few things to check here:

• Do the confidence intervals overlap? If they do, you don’t yet have strong evidence that one ad is truly better — the difference could be noise.

• What’s the relative lift? A jump from 4.20% to 5.10% is a 21% relative increase, even though it looks small in absolute percentage points. Relative lift is often what matters for budget conversations.

• Is the sample size large enough? Small sample sizes can produce large, misleading swings. If your test hasn’t hit its pre-planned sample size, be cautious about calling a winner.

Step 5: Diagnose why one ad won

This is where the funnel view becomes useful again. Say Ad B won on acquisition rate — look back at each step to understand the story:

• If start rate jumped the most, Ad B’s messaging may have set clearer expectations, attracting visitors who were more ready to apply.

• If submit rate jumped the most, Ad B may have attracted a more qualified audience less likely to abandon the form.

• If acquisition rate from submissions jumped, the difference might be about audience quality (e.g., credit profile) rather than the ad’s messaging at all.

This diagnosis matters because it tells you what to repeat next time, not just which ad to keep running.

Common pitfalls to avoid

• Peeking too often and stopping early. Checking results daily and stopping the moment Ad B looks better inflates your false-positive rate. Stick to your pre-planned sample size or duration.

• Ignoring external events. A holiday, a rate change, or a competitor promotion can shift acquisition behaviour independent of your ads. Try to test during comparable periods, or account for known events in your read of the results.

• Comparing rates from different audiences. If Ad A and Ad B were shown to systematically different audiences (not randomly split), any “winner” is confounded and not trustworthy.

• Over-optimizing for the wrong step. Optimizing purely for “starts” or “clicks” can inflate top-of-funnel numbers while acquisition rate stays flat or drops — always tie back to the north-star metric.

• Ignoring statistical significance. A/B tests need enough data to be conclusive. A 5% lift on a tiny sample is not the same as a 5% lift on a robust one.

Takeaways

• Define the funnel first: Visitors → Started Application → Submitted Application → Acquisition.

• Pick one primary metric — Acquisition Rate = Acquisitions ÷ Visitors and let secondary metrics explain the “why” not decide the “which.”

• Plan sample size and test duration in advance, and don’t call a winner until you get there.

• Use confidence intervals, not just point estimates, to judge whether a difference is real.

• Always follow up a “winning” ad with a funnel-level diagnosis — it’s the difference between a lucky test and a repeatable playbook.

A/B testing for acquisition isn’t about running more tests — it’s about running tests you can actually trust, and reading them the same disciplined way every time.

*All figures used in this article are illustrative example data created for explanatory purposes only, and do not represent the performance of any real company, product, or campaign.