How Analysts Make Sure Non-Functional Requirements Are Actually Right

Your acceptance criteria prove the payment API returns the right response. They say nothing about whether it returns it in under 800 milliseconds under load, and that gap is where production incidents live. Here is how a BA writes and tests non-functional requirements from the same API layer.

Every PAIN.001 acceptance criterion I have ever reviewed is functional. The system accepts a valid credit transfer. The system rejects an invalid IBAN. The system returns a PAIN.002 with the right status code. Sign off, close the story, move on.

Then the payment goes live, volume ramps to production levels, and the same API that passed every test starts timing out at 1,400 milliseconds because the sanctions screening service was never tested under concurrency. Nobody wrote that criterion. It was in the requirements, technically, buried in a non-functional requirements document that a solution architect wrote eighteen months ago and nobody has opened since.

This is the split that quietly defines whether a payments release survives contact with production. Functional requirements say what the system does. Non-functional requirements, the NFRs, say how well it does it: latency, throughput, availability, how it behaves when a dependency is slow. Analysts test the first category obsessively and the second one almost never, because functional testing feels like the BA’s job and NFR testing feels like someone else’s.

It is not someone else’s. And the same API layer you already use to test functional behaviour will test the non-functional side too, if you know what to send it. I break the full requirements-writing side of this down in From Vague BR to Functional Requirements, but the testing half is what almost nobody does, so that is what this article is about.

Why the split gets ignored

The honest reason NFRs go untested is that they are hard to write as pass-or-fail statements, and analysts avoid what they cannot phrase cleanly.

“The system accepts a valid PAIN.001” is trivially testable. You send one, you check the response, done. But “the system performs well under load” is not a test, it is a wish. What load? Performs how? For how long? Until someone turns that into “the payment API returns a PAIN.002 within 800ms at the 95th percentile while sustaining 50 requests per second for 10 minutes,” there is nothing to test against, and most NFR documents never get that specific.

So the criterion stays vague, vague criteria cannot be tested, and untested criteria fail in production. I watched this exact chain play out on a real-time payments rail. The functional testing was immaculate. The first busy Monday, the p99 latency blew through the Interac timeout and payments started getting auto-returned as PACS.004s, not because anything was functionally wrong but because nobody had ever measured the API under a realistic Monday.

The lesson I took from that: an NFR you cannot express as a number with a threshold and a duration is not a requirement, it is a hope. Half the job of testing NFRs is forcing them to be specific enough to test at all.

The same request, tested two ways

Here is the reframe that makes this tractable. You do not need a separate discipline for NFR testing. You need to take the functional request you already have and ask a second set of questions about it.

Take one PAIN.001 POST. The functional test asks:

Functional:  did it return the right thing?
             - HTTP 202 accepted
             - PAIN.002 status = ACSP
             - correct EndToEndId echoed back

The non-functional test asks the same request:

Non-functional: did it return well?
                - under 800ms at p95?
                - still under 800ms at 50 req/s?
                - what happens when screening is slow?
                - does it stay up for 10 minutes of load?

Same endpoint. Same payload. Two completely different questions. Most analysts only ever ask the top one, and the entire bottom block is where the incidents come from.

The tools you already use bridge this more easily than people expect. A Postman or Bruno collection that sends a functional request is one iteration count away from a load test. If you are not yet fluent in reading and writing against an API spec at that level, that is the foundation everything here sits on, and I cover it end to end in API Documentation from Scratch. The same runner that validates a flow functionally will hammer it for throughput with almost no changes.

Latency is a distribution, not a number

The single most common mistake I see analysts make when they finally do test performance: they send ten requests, look at the average, see 240ms, and write “latency: acceptable.”

Average latency is nearly useless for a payment API. The average hides the tail, and the tail is what breaks. If 95 requests return in 200ms and 5 return in 3,000ms because they hit a cold sanctions-screening cache, your average is a comfortable 340ms and your customers are experiencing timeouts on one payment in twenty.

You measure percentiles, not averages:

p50  (median)   200ms   -- the typical experience
p95             780ms   -- the threshold you actually commit to
p99            1,340ms  -- the tail that generates the incident tickets
max            3,100ms  -- the cold-cache outlier that returns a PACS.004

The p95 is usually the number in the contract, because it is the honest one. It says nineteen out of twenty payments clear under the threshold. The p99 is the one that tells you whether you are about to get paged. When I review an NFR now, the first thing I ask is which percentile the threshold applies to, because “800ms” with no percentile attached is meaningless and I have seen teams argue for a week because one side meant average and the other meant p99.

A quick way to get percentiles out of a collection run without any special tooling: capture each response time, sort the array, and index into it.

// after a load run, times[] holds every response duration in ms
const sorted = times.sort((a, b) => a - b);
const p = (n) => sorted[Math.floor(sorted.length * n / 100)];

console.log(`p50 ${p(50)}  p95 ${p(95)}  p99 ${p(99)}  max ${sorted.at(-1)}`);

That is the whole trick. Analysts think percentile analysis needs a performance-engineering suite. It needs a sorted array.

The NFRs nobody writes down

Latency and throughput get at least a token mention in most NFR documents. These four rarely do, and each one has ended a release I worked on.

Behaviour under a slow dependency. Not a dead dependency, a slow one. What does the payment API do when sanctions screening takes 6 seconds instead of 200ms? Does it queue, time out cleanly, or hold the connection open until the whole thread pool is exhausted and healthy payments start failing too? This is the failure mode that turns one slow service into a full outage, and it is almost never in the acceptance criteria. You test it by pointing your request at a stubbed screening service with an injected delay and watching what the API does.

Idempotency under retry. If a client times out and resends the same PAIN.001 with the same EndToEndId, do you process one payment or two? In payments this is not a nicety, it is the difference between a clean release and a duplicate-debit incident with real money attached. The test is simple and brutal: send the identical request twice, fast, and assert exactly one payment was created. A collection runner is perfect for this, and the same chained-validation setup I cover in Automate Kafka Validation with Postman is what I use to fire the duplicate and check the downstream topic for a single message. I have seen this criterion missing from stories for a system whose entire purpose was moving money.

Graceful degradation. When the Kafka broker for the async leg is unavailable, does the synchronous API still accept and park the message, or does the whole endpoint return 503? One of those is a degraded service, the other is an outage, and the requirement should say which one is acceptable.

Recovery. After a dependency comes back, does the backlog drain automatically or does someone have to intervene at 2am? “Recovers automatically within 5 minutes of dependency restoration” is a testable NFR. “Is resilient” is not.

Slow dependency   ->  does one slow service become a full outage?
Retry             ->  does a resend become a double debit?
Broker down       ->  degraded service or hard 503?
Recovery          ->  auto-drain or human at 2am?

None of these four is exotic. Every one is testable from the same API layer you already have a collection for. And every one, left untested, has a specific production incident with its name on it. The reason they get skipped is not difficulty, it is that they were never written as criteria, which loops back to the real problem: NFR testing fails at the requirements stage, not the testing stage.

Make the NFR a story, or it will never be tested

The fix that actually moved the needle for me was procedural, not technical. I stopped treating NFRs as a separate document and started attaching a non-functional acceptance criterion to the functional story itself.

Before, a story read:

As a corporate client, I can submit a credit transfer via the payment API. AC: Given a valid PAIN.001, the API returns 202 and a PAIN.002 with status ACSP.

After:

AC1 (functional): Given a valid PAIN.001, the API returns 202 and a PAIN.002 with status ACSP. AC2 (non-functional): The above holds at p95 < 800ms while sustaining 40 req/s for 10 minutes. AC3 (non-functional): Given screening latency of 5s, the API times out cleanly at 3s and returns a retryable status, without degrading other in-flight payments.

The moment AC2 and AC3 live in the same story as AC1, they get estimated, they get built, and they get tested, because they are in front of the team instead of in a document nobody opens. That single habit change did more for release stability than any tool I introduced. The developers started designing for the slow-dependency case because it was staring at them in the story, not discovered in production.

This is also the fastest way I know for an analyst to earn technical credibility, because writing AC3 correctly requires you to understand thread pools, timeouts, and failure propagation, and once you are writing criteria at that level, developers stop treating your stories as translation and start treating them as design. It is one of the skills I lean on hardest, and I go deep on this kind of technical criteria-writing in The Technical Skills Guide for BAs.

Where this is heading

The part of NFR testing that is genuinely tedious, generating realistic load profiles, injecting dependency delays, sweeping through percentile thresholds, is exactly the kind of repetitive, rule-shaped work that an agent handles well, as long as the human still owns the thresholds and reads the results. I have started using that approach for the load-generation side specifically, and it is the subject of AI Agents at Work for Analysts, The Complete Guide if you want to see where testing like this goes next. The judgement stays with you; the grinding does not.

But do not start there. Start by adding one non-functional acceptance criterion to your next story. Pick the latency one, because it is the easiest to phrase as a number. Get it estimated and tested alongside the functional criteria. Then add the slow-dependency one, because that is the one that prevents actual outages.

The analysts who get trusted with the hard releases are not the ones who wrote the most thorough functional tests. Everyone writes those. They are the ones who tested the how well, caught the p99 problem in SIT instead of on the first busy Monday, and turned a vague NFR document into criteria the team actually built against. Functional testing proves the payment works. Non-functional testing proves it will still work at nine on a Monday morning, and that is the test that keeps you out of the incident bridge.

Everything I build for analysts working this way lives at The Tech BA Toolkit, and if you would rather get the whole technical-BA library in one place, the Complete Tech BA Bundle collects it.

Ahmed is a Senior Technical Business Analyst with 10+ years in banking and payments. He builds practical guides and tools for analysts at The Tech BA Toolkit.