Metamorphic Testing: Testing When You Don’t Know the Answer

Photo by Leeloo The First: https://www.pexels.com/photo/question-mark-on-crumpled-paper-5428826/

One of the most persistent challenges in AI testing is determining the correct output. When systems analyse massive datasets, learn from patterns humans cannot easily verify, or produce probabilistic results, traditional expected-value testing quickly breaks down.

Metamorphic testing (MT) offers a powerful alternative. Instead of asking “Is this output correct?”, it asks a different question:

“Do the relationships between outputs behave as they should?”

What Is Metamorphic Testing?

Metamorphic testing is a technique that generates new test cases from an existing test case (the source test case) by applying a metamorphic relation (MR).

A metamorphic relation describes how changes in input should affect the output, based on properties of the function being tested.

Once a source test case has been executed, one or more follow-up test cases are created by modifying the inputs according to the MR. The expected results of these follow-up tests are not absolute values but rather relationships relative to the original result.

This makes MT particularly valuable when a traditional test oracle is missing, expensive, or impossible to define.

A Simple Example

Imagine a program that calculates the average of a list of numbers.

You run a source test case with a specific set of values and verify the average. Now consider what you know about the average function:

• If you reorder the numbers, the average should stay the same
• If you multiply every number by two, the average should also double

These are metamorphic relations.

Using them, you can generate follow-up test cases without recalculating the expected result from scratch. The relationships alone are enough to verify correctness.

From a single source test, you can generate dozens — or even hundreds — of follow-up tests that validate consistent behaviour.

When Expected Results Change

Not all metamorphic relations preserve the same output. Some deliberately change it in predictable ways.

Continuing the average example:

• Multiply all inputs by a constant → output should scale by the same constant
• Add a fixed value to all inputs → output should increase by that value

Each of these relationships allows testers to generate new tests with confidently predictable outcomes, even when absolute correctness is hard to verify.

Why Metamorphic Testing Is So Useful for AI

AI systems often operate in domains where:

• ground truth is unavailable or delayed
• outputs depend on complex statistical inference
• internal decision logic is opaque
• traditional oracles are impractical

Metamorphic testing sidesteps these problems by focusing on invariants and trends rather than exact answers.

In AI contexts, MT has been applied to:

• image recognition
• search engines
• route optimisation
• voice recognition
• recommender systems
• actuarial and predictive models

For example, if an AI system predicts life expectancy based on lifestyle data, it may be impossible to know whether any single prediction is “correct”. But it is reasonable to assert that increasing cigarette consumption should not increase predicted lifespan. That relationship becomes the oracle.

When Even the Source Test Isn’t Known to Be Correct

One of the most potent aspects of metamorphic testing is that it does not require the source test case to be correct by definition.

In situations where the function is too complex for human verification — as is often the case with AI — MT can still be applied. Testers generate multiple related test cases and verify that the relationships among their outputs hold.

Even if no individual result can be confirmed as correct, consistency across related tests builds confidence in the system.

Strengths of Metamorphic Testing

Metamorphic testing stands out because it:

• works without a traditional test oracle
• is intuitive once the domain is understood
• scales well from a single source test
• applies to both functional and non-functional testing
• is effective at revealing defects

Research suggests that just three to six well-chosen metamorphic relations can uncover the vast majority of defects detectable using traditional oracle-based techniques.

Automation and Tool Support

Follow-up test cases can be generated automatically when metamorphic relations are well specified. While commercial tools are still rare, the technique is gaining traction.

Google, for example, uses automated metamorphic testing to test Android graphics drivers via the open-source GraphicsFuzz tool. This demonstrates MT’s value even in highly technical, low-level domains.

A Practical Exercise

To practise metamorphic testing:

• Identify an AI-based application or complex program
• Derive several metamorphic relations, including:
• relations that preserve output
• relations that predictably change output
• Create one or more source test cases (even if correctness cannot be guaranteed)
• Generate follow-up test cases using the relations
• Run the tests and verify that the relationships hold

The focus is not on perfection, but on consistency.

A Closing Thought

Metamorphic testing reflects a quiet but profound shift in how we test complex systems. When answers are uncertain, relationships become anchors. When certainty is impossible, structure still exists — if we know where to look.

For AI testers, metamorphic testing is not just a clever technique. It is a way of restoring confidence in systems that resist simple verification by testing the shape of correctness rather than its exact coordinates.