Making Metrics Meaningful in LLM

I started out with what seemed like a straightforward mission: Automate the heck out of grant writing.

Simple in theory. Nonprofits have a mountain of repetitive paperwork. We had documents, we had AI, and we had a clear goal ,let the machine handle the tedious work so humans could focus on changing the world, not on Ctrl+C/Ctrl+V.

Fast-forward a few weeks, and I was sitting in front of my laptop at 2 AM, staring at a spreadsheet full of scores like:

Relevancy: 0.571  
Faithfulness: 0.777  
Context Recall: 0.5  
Output Similarity: 0.782

I had no real sense if the app was actually getting better. And that’s when it hit me, I didn’t even know what success was supposed to look like.

🧱 Phase 1: Naïve Beginnings

The first version of the app was basic: upload a few grant PDFs, generate draft answers with GPT, and let nonprofits fill in the blanks. We used vector search to grab relevant context. It worked… reasonably well. But “reasonably well” isn’t measurable.

So I turned to classic NLP metrics. You know the ones:

• BLEU: Used in translation. Counts overlapping n-grams.
• ROUGE: Similar, but recall-based.
• METEOR: Adds synonym matching and alignment.

I applied these metrics to our generated answers, comparing them to the “ideal” human-written ones.

Result? The app scored decently. But when I manually read the outputs, they were often technically similar but semantically broken or worse, hallucinated facts wrapped in grant-sounding fluff.

🔁 Phase 2: Feedback Loops (Or Lack Thereof)

The next idea was simple: ask users.

We built a simple thumbs-up/thumbs-down system with an option to leave comments. Our nonprofit partners, understandably busy with their own missions, provided feedback that was sometimes delayed, brief, or very high-level (“this is amazing” / “this missed the point entirely”) making it a bit challenging to directly translate into model improvements.

To bridge the gap, we created internal scoring matrices and introduced a “compliance checklist” for each question to structure the feedback more objectively.

Even then, it quickly became clear: relevance is subjective. What felt “spot-on” to one person might seem “completely off” to another. Navigating these differences made the metric landscape a lot more complex than we first expected.

🧪 Phase 3: The Frankenstein Metric Sheet

Eventually, we tried to measure everything:

I spent weeks fine-tuning this massive sheet, some metrics contradicted each other. High recall but low precision? Great, now we know more wrong things.

🌅 What Finally (Sort of) Worked

After breaking my brain over metrics, here’s what I’ve come to accept:

• LLMs are not search engines. You need to judge both the process (retrieval) and the product (generation).
• “Correctness” must include multiple dimensions: factual accuracy, contextual alignment, and human-readability.
• We use LLMs to evaluate LLMs. Meta, but useful. Comparing model answers to human ones using an evaluator LLM gave us meaningful semantic similarity scores that weren’t fixated on word overlap.
• Redundancy in context isn’t always bad. But it kills precision, so we introduced a context compression step before generation.

Eventually, we collapsed it into 3 scores:

• Relevance (Did you find the right stuff?)
• Faithfulness (Did you lie?)
• Similarity (Did you write like a human?)

Each one is scored via a combo of model judgment and human heuristics. It’s not perfect, but it feels right and most importantly, it gives us a way to improve week by week.