How to Build a Robust API Test Framework with Rest Assured and TestNG (Guide for Scalable Automation)

Learn how to design a scalable API test automation framework using Rest Assured and TestNG, including project structure, reusable components, data-driven testing, CI/CD integration, and best practices to reduce flaky tests and improve reliability.

We tried to build a “clean” API test framework with Rest Assured and TestNG. It worked — until CI, scale, and parallel execution exposed everything we missed. This is what actually broke and what we changed.

Not a Medium member? Drop a comment and you will get the free access link

How our “robust” Rest Assured + TestNG framework started failing at scale

The framework looked solid. CI said otherwise.

Locally, everything passed.

CI pipeline: ~31% failure rate on certain branches.

Not consistent. Not reproducible.

Just enough red builds to slow every release by 1–2 hours.

We called it “infra instability” for a week. Maybe two.

We thought structure would save us

We spent time designing the “right” framework.

• BaseTest
• RequestBuilder
• ResponseValidator
• Utils (of course)

It looked clean.

Too clean.

Turns out, abstraction hides problems really well.

The BaseTest class quietly became a god object

Everything flowed through it.

Authentication. Config. Headers. Logging. Data setup.

At some point, even retry logic.

public class BaseTest {

protected RequestSpecification request;

@BeforeMethod
    public void setup() {
        request = given()
            .baseUri(Config.get("baseUrl"))
            .header("Authorization", TokenManager.getToken())
            .contentType(ContentType.JSON);
    }
}

Looks harmless.

But TokenManager.getToken() wasn’t.

It cached tokens globally. TestNG ran tests in parallel.

You can guess the rest.

Parallel execution didn’t break the framework. It revealed it.

Before parallel:

████████████ 18 min

After:

██████ 9 min (but chaotic)

Failures jumped from ~5% to ~27%.

Same tests. Same code.

Different timing.

Requests overlapped. Test data collided. Tokens expired mid-suite.

Parallel didn’t introduce bugs. It removed the illusion that things were fine.

DataProviders felt elegant. Until they weren’t.

We leaned heavily on TestNG DataProviders.

One test. Multiple datasets. Clean.

Except:

• some datasets depended on state
• some modified shared resources
• some failed only in combination

We didn’t notice because we never ran them together locally.

CI did.

That’s where things got weird.

The request builder abstraction slowed us down

We created a reusable builder:

public class RequestBuilder {
 public static RequestSpecification getRequest() {
        return given()
            .baseUri(Config.get("baseUrl"))
            .header("Authorization", TokenManager.getToken())
            .log().all();
    }
}

Nice, right?

Except debugging became harder.

We couldn’t tell:

• which test mutated the request
• where headers changed
• why logs didn’t match expectations

We centralized logic. And lost visibility.

CI/CD exposed timing problems we didn’t design for

Locally, tests ran fast enough that timing didn’t matter.

CI added:

• network latency
• container startup delay
• shared environment contention

Suddenly:

• some APIs returned stale data
• some took longer than expected
• some just… didn’t respond in time

We increased timeouts.

Failures dropped from 31% → 14%.

But builds got slower.

And less honest.

Retry logic felt like a fix. It wasn’t.

We added retries for flaky tests.

Just 1 retry.

Failure rate dropped.

Green builds increased.

Confidence did not.

Because now we had tests that:

• fail once
• pass next time
• and never explain why

We hid instability instead of fixing it.

Something changed when we stopped optimizing for reuse

This part was unexpected.

We started duplicating small pieces of logic.

• test-specific request setup
• inline data creation
• explicit assertions

Less reusable. More obvious.

Tests became longer.

But easier to reason about.

That trade felt wrong at first.

Then… less wrong.

The framework is smaller now. Somehow better.

We removed:

• generic validators
• some shared utilities
• half the “smart” abstractions

Kept:

• minimal base config
• explicit test intent

Flaky rate dropped to ~9%.

Still not zero.

Still not predictable.

The part that doesn’t feel finished

Some tests still:

• fail only in CI
• pass on rerun
• depend on execution order (even now)

We don’t fully understand why.

We suspect environment. Or data. Or both.

Or something else.

Hard to say.

Reflection (and a bit of discomfort)

We started by trying to build a “robust framework.”

What we built first was a complex one.

Then we spent time removing complexity we introduced ourselves.

There’s a contradiction here.

Abstraction helped us scale structure. But hurt our ability to debug reality.

Even now, I’m not sure where that balance is.

It works. Until it doesn’t.

The pipeline is mostly green now.

Deployment delays dropped.

But every time a test fails without a clear reason, it still feels fragile.

Like the framework is holding… but not stable enough to trust blindly.