How a Weekend Hack Became Our Four-Year Experimentation Backbone

The story of Mendel Framework, an A/B testing and feature-flag platform we built in-house and just put on GitHub under the MIT license.

Experiment dashboard

The startup math problem

Rewind four years.

We were an early-stage startup with the usual constraints. A small engineering team. A product roadmap longer than the runway. Customers who did not tolerate broken releases. And a finance spreadsheet where every recurring SaaS line item got triple-checked before anyone hit approve.

Underneath that sat a quieter pressure. We had to ship features quickly, and we had to ship them carefully. Was the new checkout flow actually converting better, or did we just feel good about it on demo day? Should the redesigned billing screen go to everyone on Monday, or to 5% of enterprise accounts first? What happens if a feature works for US users but tanks engagement in EMEA?

The product team had the right instinct: bring in a third-party experimentation platform. LaunchDarkly, Optimizely, GrowthBook. Pick one, plug it in, get on with the roadmap.

Two things stopped us:

The first was cost. A SaaS line item for experimentation, at our stage, meant cutting something else. Nothing else was cuttable.

The second was engineering bandwidth. Adopting a third-party tool isn’t free either. There’s integration work, a learning curve, schema changes, SDK upgrades, and on-call implications. Engineering bandwidth was the scarcest resource in the building.

So we built our own.

A week. That’s all v1 took.

I remember the conversation. The PM team needed four things.

A way to define a new experiment without filing an engineering ticket. A way to bucket users into variants deterministically, so the same user always saw the same experience. A way to read metrics and graduate the winner or kill the loser. Targeting rules over country, plan tier, and cohort, so we could ship the right experiment to the right audience.

We sat down on a Monday. By Friday, the first version of what we now call Mendel Framework was running in production.

It wasn’t complete. We punted on layered experiments, which let you run multiple mutually exclusive experiments on the same audience without contamination. It sounds simple. It isn’t. We needed to ship something.

It was enough. The PM team could define experiments through an internal admin surface, watch them run, attribute metrics, and decide thumbs-up or thumbs-down. No SaaS bill. No vendor lock-in. No data leaving our VPC.

Four years of incremental growth

The framework grew up alongside the company.

Layered experiments arrived next, with global holdouts, so the PM team could finally run checkout-redesign and new-billing-flow at the same time without one polluting the other. The holdout slice let us measure cumulative lift across everything running in a layer.

Then prerequisites. Gate experiment B on experiment A’s variant, so funnel experiments could compose.

Then force-assign overrides, because every QA engineer, every customer-success rep, and every “can you put this enterprise account on the new flow for the demo?” request eventually arrives at your door.

Then variant payloads. A flag was no longer just on or off; it carried a typed JSON blob the application code could consume.

Then deterministic bucketing via FNV-1a hashing, so a client SDK could evaluate the same flags as the server without a round-trip, and a user’s variant assignment stayed stable across servers, regions, and deploys.

Then exposure logging that streamed into the analytics warehouse for downstream metric attribution.

Then a TTL cache on the hot read path, because experimentation cannot add latency.

And finally a React admin UI, so PMs, customer success, and growth could manage experiments without pinging engineering.

None of this came from a roadmap. Every feature came from a real ask, a real escalation, or a real “wait, this is a footgun” moment. The shape of the framework is the shape of four years of internal feedback compressed into code.

Why Mendel

Gregor Mendel was the 19th-century Augustinian friar who founded modern genetics. He spent eight years breeding pea plants in a monastery garden, running controlled experiments on inherited traits, and writing down the results.

His work is the original blueprint for product experimentation: controlled and reproducible. We wanted that rigor in our stack. The name stuck.

What you get

The framework we’re open-sourcing is the same code that’s served us in production for four years, generalised so it carries no business concepts. You bring the attributes; it does the bucketing.

Admin dashboard

Running, graduated, and failed counts. A breakdown of A/B tests, feature flags, and active layers. A jump table for what’s live right now.

Experiment list

Filter by name, state, status, type, or environment. Rollout, variant count, layer membership, and active state are all visible at a glance.

Experiment detail

A single page covers everything about an experiment: status, rollout, variant weights, payloads, targeting rules, layer assignment, and the live list of enrolled items. Force-assign and per-item overrides sit on the same screen.

Experiment form

A guided form for new experiments. Rollout type, salt, dates, variants with JSON payloads, targeting rules, prerequisites, layer assignment. This is the surface the PM team lives in.

Layers and holdouts

Group experiments into a layer to enforce mutual exclusion across audiences. Carve out a global holdout slice to measure cumulative lift from everything in the layer.

Under the hood

Bucketing is deterministic. The framework hashes salt:item_id with FNV-1a and divides by 2^32 to get a value in [0, 1). Same input, same output, every time. No coordination is needed between the backend and any client SDK.

There are two rollout modes. A_B_TESTING does probabilistic weighted variant bucketing. FEATURE_FLAG does explicit per-item enrollment.

Targeting supports eq, in, gt(e), lt(e), contains, starts_with, ends_with, regex, exists, and a few others, combined with all or any semantics. Layers enforce mutual exclusion across overlapping audiences, with an optional global holdout slice. Prerequisites gate one experiment on another's variant assignment. Force-assign pins a specific item to a specific variant for QA, demos, and the 11pm customer escalation.

Variant payloads carry arbitrary JSON alongside each variant: copy strings, config blobs, nested feature toggles. Exposure and audit hooks stream every evaluation and every mutation into whatever pipeline you use (Segment, Amplitude, BigQuery). A TTL cache on the hot read path keeps the latency budget intact.

Express integration ships drop-in client and admin routes, with optional celebrate validation. The React admin UI gives non-engineers a full surface to manage experiments. docker compose up produces a working stack in under a minute.

A short taste of the API:

const { service } = createMendelFramework(mongoose, {
  generateId  : uuid,
  environment : 'prod',
  cache       : { enabled: true, ttlMs: 5000, max: 1000 },
});

await service.createExperiment({
  exp_name      : 'exp_new_checkout',
  roll_out_type : ROLL_OUT_TYPE.A_B_TESTING,
  roll_out_value: 80,
  variants: [
    { key: 'control',   weight: 50, payload: { ui: 'classic' } },
    { key: 'treatment', weight: 50, payload: { ui: 'streamlined' } },
  ],
  targeting: {
    match: 'all',
    rules: [
      { attribute: 'plan',    op: TARGETING_OP.IN, values: ['pro', 'enterprise'] },
      { attribute: 'country', op: TARGETING_OP.EQ, values: 'US' },
    ],
  },
}, { id: 'admin' });
const result = await service.evaluate('exp_new_checkout', 'USER_42', {
  plan: 'enterprise', country: 'US',
});
// → { variant: 'treatment', reason: 'bucketed', payload: { ui: 'streamlined' } }

That is the full integration surface. No SDK install. No vendor account. No outbound network call. Everything runs against your own Mongo, behind your own auth, in your own VPC.

Why open-source it

Because we remember being the team that couldn’t afford the SaaS.

We remember staring at pricing pages and doing back-of-envelope math. Two months of cloud credits, or half a junior engineer’s salary, or the gap between extending runway and not. We knew we needed experimentation discipline to make good product decisions. We also knew we couldn’t pay for it.

Four years later, the framework that got us out of that bind still runs the same job every day. Keeping it private feels selfish.

So it’s on GitHub under MIT. The core logic, the Express integration, the React admin UI, seed data, tests. The whole thing.

If you’re an early-stage startup that needs experimentation discipline but can’t yet justify a SaaS subscription, this is for you. Clone it, run docker compose up, run npm run seed, and you'll have a working stack on your laptop in under a minute.

If you outgrow it later and migrate to a hosted platform, that’s the right path. You shouldn’t have to skip the experimentation discipline phase entirely because the tooling was out of budget.

If the framework helps you ship a safer experiment, star the mendel-framework repo on GitHub. Mendel Framework is MIT-licensed. The quickstart, docs, and a Dockerised demo stack are all in the README.