Our Journey Optimizing LLM-as-Judge Evaluation Prompts

How we used GEPA and ACE to triple our evaluation accuracy for legal document updates

Prompt optimization isn’t just about better wording — it’s about systematic evolution.

The Challenge: When LLMs Judge LLMs

We had a problem. Our system generates updates for legal documents — think regulatory changes that need to ripple through thousands of practice notes. But how do you know if the generated updates are actually good?

Enter LLM-as-Judge: using an LLM to evaluate another LLM’s outputs. Simple in theory, deceptively hard in practice.

Our first attempt? 38% correlation with human experts. Basically a coin flip with extra steps.

This is the story of how we got to 81% — and the two algorithmic breakthroughs that got us there.

The Evaluation Dimensions

We needed to evaluate generated legal updates across three key dimensions:

Dimension → Completeness Question → Does the update cover all necessary information? Why It’s Hard → Requires understanding what should be included

Dimension → Correctness Question → Is the update factually and legally accurate? Why It’s Hard → Requires domain expertise

Dimension → Position in Section Question → Is the update placed appropriately? Why It’s Hard → Requires document structure understanding

Each dimension had a 3-point scale, and we had ~300 expert-annotated examples to train against. For each example, we had three versions of a document section:

• Original section: The document before any updates • Ground truth section: The section as edited by subject matter experts (SMEs) • Generated section: Our LLM’s attempt at the update

The evaluation task: determine how well the generated update matches the quality of the expert edit.

Phase 1: The Handcrafted Baseline (38%)

Like most teams, we started with prompt engineering intuition:

{# Our first attempt - completeness_evaluation.jinja2 #}
You are an expert legal editor evaluating document updates.

Given:
- Original section: {{ original_section }}
- Ground truth (expert edit): {{ ground_truth_section }}
- Generated update: {{ generated_section }}
- Alert context: {{ alert_summary }}

Rate the completeness of the generated update compared to the expert edit:
- complete: Covers all necessary information
- partial: Covers some but not all necessary information
- incomplete: Missing significant information

Provide your rating and a brief rationale.

Results:

Results:

Completeness        →  38.2%
Correctness         →  37.6%
Position in Section →  40.8%

Not great. We tried dozens of variations — adding examples, restructuring instructions, tweaking temperature. Nothing moved the needle significantly.

Key insight: Manual prompt engineering hits a ceiling quickly. We needed systematic optimization.

Phase 2: GEPA — Evolutionary Prompt Optimization (51%)

The Algorithm

GEPA (Genetic Evolutionary Prompt Adaptation) treats prompt optimization as an evolutionary search problem:

Step 1 → EXECUTE: Run prompt on training set
Step 2 → REFLECT: LLM analyzes errors (T=0.7)
Step 3 → MUTATE: Generate 5 prompt variants
Step 4 → EVALUATE: Test variants on validation set
Step 5 → SELECT: Pareto optimization (accuracy vs cost)
         [Repeat for 4 iterations]

The Reflection Step

The magic is in the reflection. After identifying errors, we ask the LLM:

{# GEPA reflection prompt #}
You are analyzing evaluation errors to improve the prompt.

These cases were incorrectly evaluated:
{% for error in errors[:10] %}
- Case {{ error.id }}: Predicted {{ error.predicted }}, Actual {{ error.actual }}
  Context: {{ error.context | truncate(200) }}
{% endfor %}

Current prompt:
{{ current_prompt }}

Analyze the error patterns and suggest specific improvements.
What criteria is the prompt missing? What's causing systematic errors?

Results

After 4 iterations (~4 hours, ~$50–60 API cost per dimension):

Dimension              Baseline   GEPA     Improvement
─────────────────────────────────────────────────────
Completeness           38.2%     47.3%     +9.1%
Correctness            37.6%     42.1%     +4.5%
Position in Section    40.8%     61.5%     +20.7%

What GEPA discovered: • Position evaluation needed explicit structural cues (“Is this in the definitions section?”) • Completeness required comparison against specific alert elements • Correctness needed domain-specific legal accuracy criteria

Phase 3: ACE — Agentic Context Engineering (60%)

GEPA was good, but had a fundamental limitation: full prompt rewrites risk losing good insights.

The Context Collapse Problem

Iteration 1: Prompt gains insight A, B, C
Iteration 2: Rewrite keeps A, B but loses C
Iteration 3: Rewrite adds D, E but loses A
Result: Oscillating performance, lost discoveries

ACE’s Solution: Three Specialized Agents

ACE (Agentic Context Engineering) uses a 3-agent architecture with incremental updates:

Itemized Context with Metadata

Instead of a monolithic prompt, ACE maintains itemized insights:

{"context_items": [
{
"id": "C1",
"content": "For position evaluation, check if content type matches section purpose",
"helpful_count": 12,
"harmful_count": 2,
"source": "round_2_generator"
},
{
"id": "C2",
"content": "Legal updates about procedural changes belong near existing procedures",
"helpful_count": 8,
"harmful_count": 1,
"source": "round_3_generator"
}
]
}

Grow-and-Refine Strategy

PHASE 1: GROW (Rounds 1-3)
├── Generator creates diverse insights
├── Reflector tests each insight
└── Curator adds ALL items (no removal)
    → Encourages exploration

PHASE 2: REFINE (Rounds 4-6)
├── Remove items where harmful ≥ helpful
├── Deduplicate similar items (cosine > 0.85)
└── Converge to optimal context
    → Focuses on best performers

Results

Using the same seed prompts as GEPA, ACE achieved:

Dimension              GEPA      ACE      Improvement
─────────────────────────────────────────────────────
Completeness           47.3%    53.7%     +6.4%
Correctness            42.1%    54.6%     +12.5%
Position in Section    61.5%    66.4%     +4.9%

Why ACE outperforms GEPA:

• No context collapse (incremental updates)
• Per-item performance tracking
• Better exploration-exploitation balance
• ~30% cheaper ($40 vs $60 per dimension)

An Interesting Observation: Deduplication Challenges

One thing we noticed: ACE’s deduplication wasn’t as aggressive as expected. Despite using cosine similarity (threshold 0.85) to merge similar insights, the final prompts still contained noticeable repetitions.

For example, we’d end up with both:

• “Check if the update belongs in the definitions section”
• “Verify content type matches section purpose (definitions vs enforcement)”

These are semantically related but different enough to survive the similarity threshold. The optimal deduplication threshold is tricky — too aggressive and you lose nuance, too lenient and you get redundancy. This remains an open challenge for incremental prompt optimization.

Phase 4: Domain Knowledge Integration (60–81%)

The final push came from combining ACE with domain-specific refinements learned through iterative experimentation.

What We Added

1. Confusion Matrix Analysis

We built confusion matrices for each dimension to identify systematic errors:

Position in Section - Before Optimization:
                     Predicted
                     appropriate uncertain inappropriate
Actual appropriate      45          12          8
uncertain                3           2          5
inappropriate           18           4          3

Key insight: 18 "inappropriate" cases predicted as "appropriate"
→ Prompt was too lenient, needed stricter placement rules

This revealed that our prompt was over-accepting — it marked positions as appropriate even when content clearly belonged elsewhere.

2. Practice Area-Specific Patterns

Different legal domains behave very differently:

Litigation      → Updates often reference case precedents — position near case discussions
Corporate & M&A → Definition updates are common — check for definitional sections
Finance         → Regulatory citations matter — position near compliance sections
Bankruptcy      → Procedural changes belong near procedural guidance

We encoded these patterns as explicit evaluation criteria.

3. Common Error Patterns

Through error analysis, we discovered recurring failure modes: • False completeness: LLM marks updates as “complete” when they mention a topic but lack specifics • Position blindness: Definitions placed in enforcement sections rated as “appropriate” • Citation confusion: Missing statutory references not flagged as incomplete

Final Prompt Structure

{# Production evaluation prompt - position_in_section #}

You are evaluating whether a legal document update is positioned appropriately.

## Context
- Practice Area: {{ practice_area }}
- Alert Summary: {{ alert_summary }}

## Original Section (Before Update)
{{ original_section }}

## Expert Edit (Ground Truth)
{{ ground_truth_section }}

## Generated Update (To Evaluate)
{{ generated_section }}

## Evaluation Criteria
{% for item in context_items %}
- {{ item.content }}
{% endfor %}

## Domain-Specific Rules
- If practice area is "Corporate & M&A": Check for definitional vs operational placement
- If practice area is "Litigation": Verify case law references are near precedent discussions
- If practice area is "Finance": Regulatory updates belong near compliance guidance

## Your Task
Compare the generated update against the expert edit. Rate position appropriateness:
- appropriate_position: Update is logically placed given document structure
- uncertain_position: Placement is questionable but defensible
- inappropriate_position: Update clearly belongs elsewhere

Provide your rating followed by a brief rationale.

Final Results

Dimension              Phase 1   Phase 4   Total Gain
─────────────────────────────────────────────────────
Completeness           38.2%     61.0%     +22.8%
Correctness            37.6%     58.0%     +20.4%
Position in Section    40.8%     80.9%     +40.1%

The Journey Visualized

Handcrafted baseline       →  ~39% avg accuracy
After GEPA                 →  ~50% avg accuracy  (+11%)
After ACE                  →  ~58% avg accuracy  (+8%)
After Domain Knowledge     →  ~67% avg accuracy  (+9%)
Peak (Position in Section) →   81%

Key Takeaways

1. Manual Prompt Engineering Hits a Ceiling

Trial-and-error optimization plateaus around 40% for complex evaluation tasks. You need systematic approaches.

2. Evolutionary Methods Work, But Have Limits

GEPA provided +13% average improvement, but full prompt rewrites cause context collapse.

3. Agentic Architectures Are Superior

ACE’s 3-agent system with incremental updates outperformed GEPA while being cheaper and more interpretable.

4. Domain Knowledge Still Matters

The final +6–15% came from understanding our specific domain — confusion matrix analysis, practice area differences, and encoding evaluation edge cases explicitly.

5. Ground Truth Comparison is Crucial

Including the expert-edited section alongside the generated update gave the LLM a concrete reference point for quality assessment — not just “is this good?” but “is this as good as what an expert would do?”

6. Deduplication Thresholds Are Tricky

ACE’s similarity-based deduplication (cosine > 0.85) still left redundancies. Finding the right balance between preserving nuance and eliminating repetition remains an open challenge.

Reproducibility

Both GEPA and ACE are based on published research:

• GEPA: “Genetic Prompt Search via Exploiting Language Model Probabilities”
• ACE: “Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models”

What’s Next?

We’re exploring:

• Minority class detection: Current prompts struggle with rare error types (e.g., “partially correct”)
• Adaptive deduplication: Dynamic similarity thresholds based on semantic clustering
• Continuous learning: Feeding production errors back into ACE optimization loops

Have questions about implementing GEPA or ACE for your own evaluation tasks? Drop a comment below or reach out!

Tags: #LLM #PromptEngineering #MachineLearning #NLP #LegalTech #AIEvaluation