Measuring Context Engineering

Lessons learned from Building Observable AI QA Testing Workflows

Software engineering maturity is often measured through test metrics.

How often are tests run? How long do they take? Is runtime increasing or decreasing over time? How often do tests pass or fail? What is the trend of passing versus failing tests? How many defects are found in a given environment? How severe are those defects? Are the same classes of problems recurring?

Years ago, I wrote extensively about traditional software quality metrics in 64 Test Metrics for Measuring Progress, Quality, Productivity & More.

Many of those metrics still matter today. Teams still need visibility into execution trends, failure rates, defect leakage, system stability, and delivery health.

But AI-assisted engineering workflows introduce an entirely new operational layer — one that traditional software metrics were never designed to measure.

In a previous article, I explored the shift from prompt engineering to context engineering. A good prompt may help once. A good context system helps repeatedly.

This article extends that idea further.

If context engineering is becoming a real engineering practice, then we need practical metrics to measure its maturity.

We need to ask:

• Is the AI producing better outputs over time?
• Are teams spending less time correcting or steering it?
• Is the provided context becoming more reusable, reliable, and current?
• Are AI-assisted workflows improving delivery quality — or merely accelerating draft generation while preserving the same underlying inefficiencies?

In other words: We need metrics that tell us whether AI workflows are maturing — not simply whether we are using AI more often.

To explore this problem practically, I built an experimental repository:

context-engineering-metrics repository

The repository compares multiple AI workflow maturity patterns by generating, executing, and evaluating pytest API tests against the Restful Booker API.

The original goal was straightforward: Can we mathematically quantify whether increasingly sophisticated context engineering workflows actually produce better API testing outcomes than simple prompt-only agents?

I expected orchestration-heavy workflows to clearly outperform simple prompt-only systems.

That is not what happened.

The experiment revealed something far more nuanced:

• retrieval grounding often provided the highest practical ROI
• orchestration introduced measurable operational overhead
• workflow complexity created new failure surfaces
• evaluation itself became surprisingly difficult

More context, tooling, and orchestration do not automatically produce better outcomes. As AI workflows become increasingly sophisticated, operational complexity can grow faster than reliability, stability, and execution quality.

Most importantly, the repository demonstrated why AI workflows should increasingly be evaluated as observable software systems rather than isolated model interactions.

The Experiment

The repository evaluates five AI workflow architectures:

Workflow Architectures

Baseline        → Prompt-only generation
RAG             → Retrieval-augmented generation
Memory          → RAG + reusable memory
Tool            → Retrieval + executable tooling
Orchestration   → Coordinated multi-step workflow

Each workflow:

1. generates pytest API tests
2. executes those tests independently
3. collects operational metrics
4. compares workflow maturity characteristics

The workflows all target the same API system but differ in:

• retrieval behavior
• orchestration complexity
• token consumption
• tooling
• execution strategy
• workflow coordination

The generated tests include:

• health checks
• CRUD API operations
• authentication workflows
• cleanup logic
• negative authorization scenarios

The experiment eventually evolved into:

• independent test generation per agent
• independent pytest execution per agent
• execution-based evaluation
• workflow observability metrics

The Results

The initial expectation was that orchestration-heavy workflows would clearly outperform prompt-only systems.

Instead:

• all workflows initially passed execution
• orchestration consumed significantly more tokens
• measurable quality differences remained surprisingly small

Once richer pytest suites were introduced, the advanced workflows actually became increasingly fragile.

The orchestration-heavy workflows began failing due to:

• malformed generated code
• import failures
• orchestration instability
• pytest collection errors
• execution brittleness

Meanwhile, the simpler baseline workflow often remained stable because it generated fewer, simpler tests.

One of the strongest findings was that retrieval-augmented generation frequently provided the best balance between:

• grounding quality
• execution stability
• operational simplicity
• token efficiency

This suggests that: Retrieval grounding may provide the highest practical return on complexity investment for many engineering workflows.

The experiment ultimately did not prove that orchestration-based systems universally outperform simpler workflows.

Instead, it revealed a more important engineering reality: More context does not automatically produce better outcomes.

AI workflow maturity evolves from simple prompt-based interactions toward increasingly coordinated systems that incorporate retrieval, memory, tooling, execution environments, and orchestration. Each maturity layer improves capability while simultaneously increasing operational complexity, token cost, and potential failure surfaces.

The Metrics

The repository evaluates AI workflow maturity using operational metrics designed to measure workflows as executable systems rather than simple text generators.

1. Context Relevance Score

Measures whether the workflow retrieved the expected grounding information necessary for task execution.

Examples include:

• API contracts
• required request headers
• authentication requirements
• expected schema fields
• endpoint definitions

context_relevance_score =
matched_required_context / total_required_context

Where:

• matched_required_context = number of expected context items successfully retrieved
• total_required_context = total required context items defined for the task

A baseline prompt-only agent intentionally scores low because it retrieves no external grounding context.

2. Task Completion Score

Measures how completely the generated workflow output satisfies the expected task objectives.

task_completion_score =
matched_success_criteria / total_success_criteria

Where:

• matched_success_criteria = number of expected workflow behaviors satisfied
• total_success_criteria = total expected behaviors

Examples of success criteria include:

• generating pytest tests
• creating valid booking payloads
• asserting response fields
• handling authentication
• performing cleanup logic

One of the most important findings from the experiment was that naive text matching can accidentally reward prompt echoing instead of genuine workflow quality.

3. Execution Success Score

Measures whether the generated pytest tests successfully executed.

execution_success_score =
tests_passed / tests_collected

Where:

• tests_passed = number of passing pytest tests
• tests_collected = total discovered pytest tests

This became one of the most valuable metrics because it evaluates observable runtime behavior rather than generated text quality alone.

4. Tool Invocation Efficiency

Measures whether the workflow invoked the correct tools while minimizing unnecessary orchestration overhead.

tool_invocation_efficiency =
(useful_tool_calls / required_tool_calls)
− orchestration_penalty

Where:

• useful_tool_calls = correctly invoked workflow tools
• required_tool_calls = expected tools for the task
• orchestration_penalty = penalty for excessive or unnecessary workflow complexity

This metric captures an important systems-engineering tradeoff: More orchestration is not automatically better orchestration.

5. Tokens Used

Measures the approximate operational cost of the workflow.

tokens_used =
prompt_tokens +
context_tokens +
output_tokens +
tool_tokens

Where:

• prompt_tokens = task instructions
• context_tokens = retrieved grounding information
• output_tokens = generated workflow output
• tool_tokens = orchestration/tooling overhead

This metric helps quantify the operational cost of increasingly sophisticated AI workflows.

6. Retries

Measures how many regeneration or rerun attempts were required after failures.

retries =
total_execution_attempts − 1

Retries become especially important for future orchestration systems that support:

• autonomous repair
• failure diagnosis
• rerun strategies
• execution recovery loops

The current experiment primarily simulates retries, but future workflow versions aim to derive them from real repair behavior.

The Most Important Lessons Learned

This is an example result from the run of the repo and lets highlight the lessons learned.

1. More Context Does Not Automatically Produce Better Outcomes

Adding retrieval, memory, tools, and orchestration increased complexity and token cost, but did not consistently improve execution quality for simpler tasks.

This was one of the most important findings from the experiment.

As AI workflows evolve from simple prompts to orchestration-heavy systems, operational complexity increases faster than execution reliability. Retrieval-augmented workflows often provided the strongest balance between grounding quality, execution stability, and token efficiency in the experiment.

2. Retrieval Grounding Often Provided the Highest Practical ROI

RAG-based workflows frequently achieved the best balance between:

• context quality
• operational simplicity
• execution stability
• token efficiency

For moderate-complexity tasks, retrieval grounding alone often outperformed heavier orchestration pipelines.

3. Execution-Based Evaluation Is Far More Reliable Than Text-Based Scoring

Metrics based purely on generated text were surprisingly easy to mislead through:

• prompt echoing
• semantic overlap
• deterministic template reuse

Pytest execution exposed actual workflow quality.

This became one of the clearest arguments for treating AI workflows as executable systems rather than text-generation systems.

4. Workflow Complexity Introduces New Failure Surfaces

Advanced workflows became increasingly fragile, revealing issues such as:

• malformed generated code
• orchestration instability
• import failures
• execution brittleness

This created a very realistic engineering tradeoff:

Simpler Workflows
- lower capability
- lower cost
- more stable
- easier debugging

Advanced Workflows
- higher capability
- higher cost
- more fragile
- harder debugging

5. AI Workflows Should Be Evaluated as Observable Systems

The most important lesson was this: AI workflows increasingly resemble software systems, not chatbot conversations.

Meaningful evaluation requires:

• execution validation
• retry metrics
• orchestration efficiency
• observability
• runtime instrumentation
• recovery analysis

not merely prompt quality.

Final Thoughts

The experiment ultimately did not prove that orchestration-based AI systems universally outperform simpler workflows.

Instead, it revealed something more valuable: Context engineering becomes increasingly useful as workflow ambiguity, execution complexity, and recovery requirements increase.

Simple tasks often do not justify heavy orchestration overhead.

The future of AI engineering likely depends less on prompt optimization alone and more on measurable workflow maturity.