argonx: A Bayesian Decision Engine for A/B Experiments

There is a moment every data scientist knows. The experiment has run long enough. The primary metric is up. The p-value is below the threshold. And the tool you’re using — whichever one it is — hands the result back to you and calls it done.

What happens next is where things get complicated. Because a p-value tells you that the observed difference is unlikely under the null hypothesis. It does not tell you which variant to ship. It does not tell you how much you lose if you get it wrong. It does not tell you whether the effect is large enough to matter, whether a guardrail metric is quietly deteriorating, or whether the numbers you’re looking at have been distorted by novelty effects or early peeking. The statistical test has finished. The actual decision has barely started.

This is the gap I spent four months building around.

Why the Existing Tools Stop Too Early

The open source landscape for Bayesian A/B testing is thinner than it should be. There are libraries that wrap PyMC or Stan and return a posterior distribution. There are conjugate prior implementations that handle binary outcomes quickly and correctly. There are well-documented notebooks covering specific scenarios. What is almost entirely absent is a framework that treats the decision itself as the hard problem — because it is.

Inference is the tractable part. Given data and a model, computing a posterior is a solved problem. PyMC handles it well. What PyMC has no concept of is what to do with that posterior once you have it. Which variant should you ship? How much do you lose on average if you make the wrong call? Is the effect you’re seeing large enough to justify a deployment, or is it statistically real but practically irrelevant? Are all your business conditions satisfied at the same time, or are you looking at metrics independently when their correlations are what matter? Can you stop the experiment now, or will stopping inflate your false positive rate?

None of that is in PyMC. None of it is in scipy. The models are plumbing. The decision logic is where the actual difficulty lives, and it is almost entirely unaddressed by existing tools.

What a Real Decision Requires

Running a production A/B experiment means operating under several simultaneous constraints that a single metric and a single threshold cannot capture.

Most experiments involve guardrail metrics — metrics that must not degrade even if the primary metric improves. Page load time. Error rate. Churn. A variant that lifts revenue by four percent while quietly increasing error rate by one percent is not obviously a good ship. The tradeoff requires quantification, not intuition. And when the primary metric is moving in the right direction while a guardrail is moving in the wrong direction, no mathematical formula resolves that automatically. The right behavior is to surface the conflict clearly — with the full posterior picture for both metrics — and make explicit that a human judgment is required. Most tools either ignore the conflict entirely or obscure it inside a composite score that papers over the tension. Neither is acceptable.

Multi-variant experiments introduce a different problem that most frameworks handle incorrectly. If you are testing three variants and you compute P(B > A) and P(C > A) separately, those probabilities do not sum to one. You are double-counting probability space. The correct approach is to ask, at each draw from the posterior, which variant has the highest value — and tally that across all draws simultaneously. This simultaneous argmax is not a minor implementation detail. It is the only mathematically coherent way to answer the question of which variant is most likely to be the true winner when more than two options are on the table.

Expected loss is the quantity that best captures the cost of a wrong decision. If you ship a variant and it turns out to be worse than the alternative, the expected loss is the average magnitude of that error, integrated over the full posterior. A variant with a high probability of being best can still carry meaningful expected loss if the downside scenarios are severe. CVaR — Conditional Value at Risk — goes further, measuring the expected loss specifically in the worst-case tail of the distribution. A decision that looks acceptable in expectation can look very different when you examine what happens in the bottom five percent of outcomes.

Practical significance is a separate question from statistical significance, and conflating them is one of the most common errors in applied experimentation. An effect can be highly statistically significant — far from zero with very high probability — and still be too small to justify a deployment. ROPE, the Region of Practical Equivalence, defines a band around zero within which effects are considered too small to matter for business purposes, and evaluates whether the posterior credible interval lies outside it. Statistical significance without practical significance is noise.

Sequential testing is where frequentist frameworks have a fundamental limitation that Bayesian methods do not share. In a frequentist framework, peeking at results before the planned sample size is reached inflates the false positive rate in ways that are difficult to correct for. Bayesian expected-loss stopping does not have this problem. You can evaluate the stopping criterion at any point during the experiment, and stopping when expected loss drops below a threshold is always a valid decision — regardless of when you look.

argonx

argonx is a Bayesian decision engine built to handle all of this correctly. You bring your data and define what matters. The framework runs the inference, computes every quantity relevant to the decision, surfaces conflicts where they exist, and packages everything into a result you can actually act on.

The models it uses — lognormal for revenue, binary for conversion, Gaussian and Student-T for latency, Poisson for count data — are the plumbing. Each has a flat variant for homogeneous populations and a hierarchical variant for segmented data. When segments are present, partial pooling handles the thin-segment problem correctly: a segment with fifty users gets a reliable estimate because it borrows strength from larger segments through a shared prior, without collapsing differences that are genuinely real. The model selection is automatic — one additional argument and the framework detects the segmented structure and fits accordingly.

The decision engine sits on top of the models and is where the framework earns its place. Simultaneous argmax across all variants. Expected loss and CVaR computed by integrating over the full posterior. ROPE evaluated against a configurable threshold. Joint probability computed across all metrics simultaneously — not as independent checks that miss correlations, but as a single draw-by-draw evaluation of whether the full business policy is satisfied. Guardrail conflict detection that flags the tension explicitly rather than resolving it arbitrarily. Loss-based sequential stopping that is valid at any checkpoint.

The API is deliberately simple:

experiment = Experiment(
    data=df,
    variant_col='variant',
    primary_metric='revenue',
    guardrails=['page_load_ms'],
    lower_is_better={'page_load_ms': True},
    model='lognormal',
    control='control',
)

result = experiment.run()
result.summary()

Behind that single method call, the framework has fit a PyMC model, extracted posterior samples, computed all decision-relevant quantities, run the guardrail checks, evaluated the stopping criterion, and assembled a structured result. The summary surfaces lift with credible intervals, P(best), expected loss, CVaR, ROPE status, guardrail outcomes, conflict flags, and a sequential stopping signal — everything needed to make the decision, nothing that obscures it.

A Design Principle Worth Stating Explicitly

argonx does not make the decision. It makes the right decision obvious.

When a guardrail is violated while the primary metric improves, the framework does not produce a weighted composite score that buries the conflict. It flags the violation, quantifies it, and states that the tradeoff requires human judgment. When the effect is statistically real but inside the ROPE, it says so. When CVaR is elevated despite acceptable expected loss, it surfaces the tail risk. The framework’s job is to give you everything you need to decide correctly — not to decide for you.

This distinction matters more than it might seem. Experimentation at scale involves tradeoffs that depend on organizational context, risk tolerance, and strategic priorities that no statistical framework can encode. The framework should make those tradeoffs visible, not invisible.

Five Worked Examples

Understanding a framework from documentation alone is slow. Understanding it through a worked example on a problem that resembles your own is fast. argonx ships with five end-to-end examples across different domains and model types.

An e-commerce checkout redesign where conversion improves but load time degrades — the guardrail conflict case. A SaaS pricing page experiment where the sequential stopping criterion fires at week two of a planned four-week run. A clinical trial where Student-T and Gaussian models produce meaningfully different posteriors on data with outliers. A three-way matchmaking algorithm experiment where simultaneous argmax produces a different answer than pairwise comparisons would. A fintech personalisation experiment with three segments of very different sizes, where partial pooling handles the thin segment correctly and the segment summary surfaces a cross-segment conflict.

Each notebook walks through the full result object with commentary. The goal is that if your problem resembles any of these, you can be running your own experiment within an hour.

argonx crossed 1000 downloads in its first week on PyPI. v0.1.1 is stable. A v0.2.0 roadmap is ready — prior sensitivity diagnostics, what-if sample size projections, forest plots for hierarchical segment comparison — but real feedback from people running real experiments matters more than the roadmap right now.

Check it out on GitHub and PyPI .