Bayesian AB Test with Hierarchical MCMC

In the world of data-driven decisions, A/B testing is a powerful tool for comparing different versions of a product or feature. But how can we go beyond the simple yes-or-no results to understand the uncertainty ? In this post, we’ll explore how to apply Bayesian A/B testing using Hierarchical MCMC (Markov Chain Monte Carlo) to get a deeper understanding of the effects of an experiment.

About the Data

For this project, we’ll use data from Kaggle: Mobile Games A/B Testing Dataset. The dataset contains:

• userid: A unique identifier for each player.
• version: The group each player was assigned to (either gate at level 30 or level 40).
• sum_gamerounds: The number of game rounds played in the first week after installation.
• retention_1: Whether the player returned to play one day after installation.
• retention_7: Whether the player returned after seven days.
• Our goal is to model how changing the gate level impacts retention (at days 1 and 7) and engagement (measured by game rounds).

Note: to simplify process we only sample some portion of the whole data.

Bayesian Approach and Hierarchical MCMC

Instead of a simple frequentist A/B test, we use a Bayesian approach to gain richer insight into uncertainty and effect size. In a Bayesian A/B test, we estimate the posterior distribution of the key metrics (such as retention rates), allowing us to quantify how likely one version is better than the other.

We used a hierarchical model to capture potential differences between player groups while sharing information between the groups. For example, we assume the retention rates for the control and treatment groups both come from a shared distribution, but they can vary. The Markov Chain Monte Carlo (MCMC) technique is used to sample from these posterior distributions, giving us estimates of the retention rates and the difference (δ) between them.

About the Code

Please find the code here. Brief explanation about the code:

• Define Priors: Each group’s retention rate has a Normal prior, influenced by hyperparameters.
• Likelihood: The likelihood is modeled with a Bernoulli distribution to represent whether players returned or not.
• Sampling: We used pm.sample() to draw samples from the posterior, allowing us to infer the retention rates and calculate δ (difference between treatment and control).

PGM: Making Relationships Clear

A Probabilistic Graphical Model (PGM) is a great way to visualize how our variables are connected. In our model, observed data (player retention and game rounds) are influenced by group-level parameters (retention probabilities) which are in turn influenced by hyperparameters (μ and σ). Using PGMs helps illustrate the flow of dependencies in our hierarchical setup, making it easier to understand how each component contributes to the inference. As seen below, stochastic variables (random variables) are shown as white circles, observed/measured variables are represented by greyed circles, and deterministic variables are represented by dots.

Result

From the posterior distributions obtained, we focused on δ (delta), which represents the difference between the retention probabilities of the treatment and control groups. The credible interval for δ allowed us to quantify our confidence in whether moving the gate had a positive or negative impact on retention. In our results (attached as a figure), the posterior distribution of δ mostly lies below zero, suggesting that moving the gate to level 40 might have worsen retention.