What It Takes to Build an Agentic QA System That Actually Works

The Problem Everyone Ignores

Most approaches to AI test generation follow the same pattern: connect Jira, feed requirements to an LLM, get test cases. Sounds simple, right?

In practice, the output is generic, shallow, and misses the real complexity of an application.

The problem isn’t the AI model. The problem is four things most teams skip:

• Context — what you feed it and how you structure it
• Constraints — what to generate, what not to, and how much
• Instructions — one-off prompts vs reusable skill files
• Standards — consistent templates so the output is usable every time

This article breaks down each one.

The New QA Engineer Metaphor

This realization changes everything. Stop thinking about “how do I prompt an AI” and start thinking about “how would I onboard a new QA engineer.”

If a new QA joined your team tomorrow, you wouldn’t just hand them a Jira ticket and say “write tests.” You’d give them:

• An overview of the application — what it does, main pages, how users navigate through it
• Access to requirements — where to find Jira stories, BRDs, design documents
• Examples of how you write test cases — so they match your style and depth
• Information about your tech stack and framework — Cypress, Selenium, Appium, or whatever your team uses
• The element selectors and identifiers — so they can write automated tests that actually work
• Clear instructions on what format you expect — manual test cases look like this, automated tests follow this structure
• Boundaries on what to focus on — test this module, skip that one, cover these scenarios

An AI agent needs exactly the same things. The metaphor maps directly to the four pillars: context is the system knowledge, constraints are the boundaries, instructions are the process to follow, and standards are the expected format.

Before You Build Anything, Ask Yourself These Questions

Before jumping into tools or structures, sit down and think through these:

What inputs does the agent need? Where are your requirements? Are they in Jira, Confluence, Git, Figma? Are they well structured or scattered? If a new QA couldn’t find them, neither can an agent.

What outputs do you expect? Manual test cases? Automated code? Both? In what format? Selenium? Appium? Do you have a framework to follow?

What context is missing? Requirements tell the agent what to test. But does it know how your system works? That context lives in people’s heads, you need to get it into a document.

What does “good” look like? If you can’t show the agent what a good test case looks like, it will guess. And it will guess wrong. Your existing test cases are the best teacher.

What are the boundaries? Should the agent generate tests for every possible scenario or just the top 10? Should it cover performance and security or only functional? Without clear constraints, the agent either over generates or misses the point.

Will you use this again? If it’s a one-time thing, a simple prompt works. If you want repeatable results across sprints and projects, you need standards, templates, and structure.

Pillar 1: Context — The Foundation of Good Output

Most people focus on writing better prompts. Focus on providing better context instead. Here’s what makes the difference:

Acceptance criteria only — the agent generates basic happy path test cases. It covers what’s written in the story but misses edge cases, business rules, and real-world complexity. Better than nothing, but not what you’d expect from an experienced QA.

Acceptance criteria + detailed requirements (BRDs, PRDs, design docs) — now the agent understands the module better. It picks up business rules, dependencies, and edge cases that weren’t in the story. The output starts to feel more complete.

Requirements + BRDs + existing test cases — this is the game changer. The agent learns how your team writes tests — the depth, the style, the kind of edge cases you care about. It stops generating generic output and starts generating tests that feel like your team wrote them.

The final layer is always human review. The goal isn’t to replace QA engineers, it’s to give them a massive head start so they spend time refining, not writing from scratch.

What counts as context:

• App overview — what the application does, main pages, user flows
• Requirements — Jira stories, BRDs, acceptance criteria
• Existing test cases — 5–10 from the same module as reference
• Locator map — key page selectors so automated tests are executable
• Domain knowledge — quirks, edge cases, things that only the QA team knows

Without this context, the agent is a new QA on day one with no onboarding. With it, the agent is a new QA who’s been briefed thoroughly.

Pillar 2: Constraints — Telling the Agent What NOT to Do

This is the pillar most teams forget entirely. Without constraints, the agent generates everything it can think of — 50 test cases when you needed 10, API tests when you only wanted UI, edge cases for modules you haven’t built yet.

Constraints tell the agent:

• What to generate — only functional test cases, or also smoke and regression?
• What to skip — don’t generate performance, security, or load tests unless asked
• How much — top 10 scenarios, not every possible combination
• Which module — only the module mentioned in the story, not related modules
• What depth — happy path and top 5 edge cases, not every boundary value

Think of constraints as the guardrails. Context tells the agent what to look at. Constraints tell it where to stop.

A good skill file bakes constraints in so you don’t have to repeat them every time. “Generate functional test cases only. Cover happy path, top 5 negative scenarios, and top 3 edge cases. Do not generate tests for modules not mentioned in the story.”

Pillar 3: Instructions — Skill Files Over One-Off Prompts

A skill file is a markdown document that tells the AI agent exactly what to do, step by step. Think of it as a reusable instruction manual. It says:

• Read inputs from here
• Look for acceptance criteria, module name, priority
• Generate manual test cases following this template
• Generate automated test cases in this tech stack’s syntax
• Apply these constraints
• Write a log of what you did

The beauty of skill files is they’re model agnostic. The same skill file works with Claude, GPT, Gemini, or any future model. Your setup isn’t locked into any one tool.

And unlike typing a new prompt every time, a skill file ensures consistency. Every run follows the same process, produces the same format, and saves in the same location.

The difference between a prompt and a skill file is the difference between giving someone verbal directions every time vs handing them a documented process they can follow independently.

I will soon publish Part 2 to show full skill file examples.

Pillar 4: Standards — Making It Work for Everyone

If it’s just you using the system, you can get away without standards. But the moment a second person needs to use it, you need consistency.

Standards cover:

• Input format — how requirements should be structured for the agent to read them. Acceptance criteria in Given/When/Then format is a good baseline.
• Output format — what a manual test case should look like (test ID, title, steps, expected result) and what an automated test case should include (describe blocks, assertions, setup/teardown).
• Structure — where inputs live, where outputs go, where logs are saved. Every project follows the same structure so the agent navigates predictably.
• Naming conventions — how to name test case files, log files, locator maps. Consistency means any QA can find anything without asking.

Standards are what turn a one-time experiment into a repeatable system. They’re boring to create but they’re the reason it works consistently.

Common Mistakes to Avoid

Sending too much context at once. Don’t dump your entire regression suite into the agent. Send only the 5–10 test cases from the same module as the story you’re generating for. More context isn’t always better — relevant context is.

Generating from code instead of requirements. If you base your tests on the code, you’re testing that the code does what the code does — not what the requirement says it should do. That defeats the purpose of QA. Generate from requirements, validate against code later.

No constraints. Without boundaries the agent over generates, mixes test types, and wastes your time reviewing irrelevant output. Define what to generate and what to skip.

Waiting for perfect documentation. Acceptance criteria in Given/When/Then format is enough to start. You’ll learn what’s missing by running the agent and seeing where it fails.

Where This Still Struggles

This approach isn’t perfect. Here’s where it still needs work:

• Ambiguous requirements — if the acceptance criteria is vague, the output is vague
• Missing domain knowledge — the agent doesn’t know your business unless you tell it
• Changing UI — locator maps break when the frontend changes frequently
• Flaky automation — generated tests can be flaky if the app has timing or async issues
• Complex flows — multi-step flows across modules are harder to generate accurately

This is an evolving approach. Every run teaches you what input was missing and what to improve next.

The Bottom Line

Agentic QA isn’t just about finding the right AI tool. It’s about getting four things right: context, constraints, instructions, and standards. Treat the AI like a new team member — give it the same onboarding you’d give a human, and it will deliver surprisingly good results.

Is this truly agentic? Not yet. The agent still needs human-curated inputs and human review of outputs. But it’s the foundation, get the context, constraints, instructions, and standards right, and autonomy becomes possible in later phases.

Start with the context, not the tool. Define your constraints. Build reusable instructions. Set standards for the team. And remember: the foundations take time, but once they’re in place, the system works for every project, every sprint, every new requirement.

That’s the difference between using AI and building with AI.