The Fallacy of Example-Based Testing

After writing too many integration and end-to-end tests across various projects, I’ve come to an uncomfortable realisation: example-based testing, the way most teams practice it, provides confidence without completeness. We write tests for the scenarios we think of, ship the code, and then discover the scenarios we didn’t think of in production.

I’ve watched teams add more and more integration tests to catch more edge cases. I’ve seen test suites grow to thousands of tests, with CI pipelines taking hours to complete. I’ve been that engineer who wrote another end-to-end test after a production incident, trying to prevent the same bug from happening again. And I’ve seen the next incident come from a different edge case that also wasn’t covered.

Something is fundamentally limited in how we approach testing.

Understanding What Example-Based Testing Actually Does

Before I get into the limitations, let me clarify what I mean by example-based testing.

Example-based testing is any testing approach where you specify concrete inputs and verify concrete outputs. This includes unit tests, integration tests, end-to-end tests, and parity tests. You think of a scenario, you encode it as a test, you move on to the next scenario.

def test_merge_sorted_lists():
    # Arrange
    list_a = [1, 3, 5]
    list_b = [2, 4, 6]

    # Act
    result = merge(list_a, list_b)

    # Assert
    assert result == [1, 2, 3, 4, 5, 6]

This test verifies one scenario. To verify another scenario, you write another test. Your coverage of the input space is bounded by the number of examples you write.

The limitation isn’t that these tests are wrong. They’re not. They verify exactly what they claim to verify. The limitation is that we treat a passing test suite as evidence that our code handles all cases, when it’s actually evidence that our code handles the cases we thought to test.

The Imagination Bottleneck

Here’s something I didn’t fully appreciate until I started analysing test suites more carefully. The coverage of example-based tests is bounded by human imagination and human time.

Consider a function that validates email addresses. A typical test suite might include:

def test_valid_email():
    assert is_valid_email("user@example.com") == True

def test_invalid_email_no_at():
    assert is_valid_email("userexample.com") == False

def test_invalid_email_no_domain():
    assert is_valid_email("user@") == False

def test_empty_string():
    assert is_valid_email("") == False

Four tests. Four scenarios. But the input space for email validation is enormous. What about: — Unicode characters in the local part — Plus addressing (user+tag@example.com) — IP address domains — Quoted strings with special characters — Maximum length boundaries — Consecutive dots — Comments in parentheses (yes, that’s valid in RFC 5321)

You could write tests for each of these. But you’d need to know they exist first. And you’d need time to write them. Most teams stop when they feel they have “enough” coverage, which is a subjective judgement that often underestimates the actual input space.

The Slow Feedback Problem

Example-based tests, particularly integration and end-to-end tests, have a scaling problem. Every new scenario you want to cover requires a new test. Every new test adds to the execution time.

I’ve worked with test suites where: — The full integration test suite took 45 minutes to run — Engineers would run a subset locally and rely on CI to catch issues — CI queues backed up during busy periods — The feedback loop between “I think this works” and “CI confirms this works” stretched to hours

The response is often to parallelise, to add more build agents, to optimise individual tests. These help. But they’re treating the symptom, not the cause. The cause is that coverage scales linearly with test count, and test count scales linearly with execution time.

The False Confidence of Parity Testing

Parity testing deserves specific mention because it’s a common pattern that has a subtle failure mode.

The idea is straightforward: you have a legacy system and a new system, you feed them the same inputs, you verify they produce the same outputs. If they match, the new system is correct.

The limitation is that parity tests only cover the inputs you feed them. If the legacy system has a bug that the new system also reproduces, parity passes. If both systems handle a certain input incorrectly in the same way, parity passes. Parity testing verifies equivalence, not correctness.

I’ve seen teams discover, after migration, that both systems had been wrong for years. The parity tests were green the entire time.

We Already Know This Doesn’t Work

Here’s the thing: we already behave as if we know example-based testing is insufficient. We just don’t talk about it that way.

Most teams I’ve worked with do some form of exploratory testing before major releases. Some call it a game day. Some call it bug bash. Some just call it “everyone stop what you’re doing and try to break this.”

The defining characteristic of exploratory testing is that you don’t follow a test plan. You explore freely, trying things that occur to you in the moment, following hunches, combining inputs in ways nobody specified. You’re explicitly not bound by the scenarios someone thought to write down.

And it works. Exploratory testing regularly finds bugs that the test suite missed. Bugs that survived thousands of automated tests get discovered in an afternoon of unscripted poking around.

But think about what that implies. If our example-based tests were sufficient, exploratory testing would be redundant. We’d find nothing. The fact that we schedule time for humans to explore outside the test plan is an admission that the test plan doesn’t cover everything. We know the examples we wrote down aren’t complete.

We’ve institutionalised the workaround without questioning the underlying approach.

A Different Approach: Property-Based Testing

Property-based testing inverts the example-based model. Instead of specifying inputs and outputs, you specify properties that should hold for all inputs. The framework generates inputs automatically.

from hypothesis import given, strategies as st

@given(st.lists(st.integers()), st.lists(st.integers()))
def test_merge_length_preserved(xs, ys):
    # Arrange
    xs_sorted = sorted(xs)
    ys_sorted = sorted(ys)

    # Act
    result = merge(xs_sorted, ys_sorted)

    # Assert
    assert len(result) == len(xs) + len(ys)

@given(st.lists(st.integers()), st.lists(st.integers()))
def test_merge_result_is_sorted(xs, ys):
    # Arrange
    xs_sorted = sorted(xs)
    ys_sorted = sorted(ys)

    # Act
    result = merge(xs_sorted, ys_sorted)

    # Assert
    assert result == sorted(result)

Two property definitions. Hundreds of generated test cases. The framework explores the input space automatically, including edge cases you wouldn’t think to test: empty lists, single-element lists, lists with duplicates, lists with negative numbers, lists with large values.

When a property fails, the framework shrinks the failing case to the minimal reproduction. Instead of “test failed with these 500 elements,” you get “test failed with [0] and [1].”

The shift is from “verify these specific examples work” to “verify this property holds universally.” The feedback loop is faster because you’re not writing individual tests for each edge case. The coverage of the input space is broader because you’re not limited by imagination.

A Different Approach: Model Checking

For concurrent or distributed systems, there’s another technique worth understanding: model checking.

Model checking tools like TLA+ let you specify what your system should do, then exhaustively explore every possible execution path within defined bounds. For a lock implementation, the model checker will verify that no two threads can ever hold the lock simultaneously — not by testing examples, but by exploring every possible interleaving.

I’ve seen race conditions that survived thousands of integration test runs get caught immediately by a model checker. The integration tests happened to never hit the specific interleaving that triggered the bug. The model checker found it because it explores all interleavings systematically.

What I’ve Learned

I’m not saying example-based tests are useless. They’re not. Integration tests verify that components actually work together. End-to-end tests verify that user journeys complete successfully. These are valuable signals.

But I’ve learned to recognise the limitations. Example-based tests tell you that specific scenarios work. They don’t tell you that all scenarios work. The difference matters.

Here’s what I’m experimenting with: — Property-based testing for functions with complex logic, data transformation, parsing, and serialisation — Model checking for concurrent code and state machines where correctness is critical — Reserving integration and end-to-end tests for high-value user journeys rather than edge case coverage — Treating example-based tests as documentation of known scenarios, not proof of completeness

I haven’t figured out the right balance yet. Property-based testing requires thinking in terms of invariants, which is a different skill than thinking in terms of examples. Model checking has a learning curve and works best for specific types of problems. Every team and codebase has different needs.

But I’ve become increasingly sceptical of test suites that grow endlessly with each new edge case discovered in production. If your tests only catch the bugs you already know about, they’re not finding bugs — they’re documenting history.

For those of you managing large test suites: have you experimented with property-based testing? How do you balance the coverage benefits against the learning curve? I’d be interested to hear what’s working for others.