LLM Evaluations Primer

Large Language Models are improving at a breathtaking pace, but evaluating them is a complex, multi-dimensional challenge. To survey this landscape, let’s zoom out for the big picture — and then zoom into the details.

Bird’s-Eye View: Four Dimensions of Evaluation

1. Technical Benchmarks — Do models solve well-defined tasks correctly?
2. User-Facing Evaluations — Are models helpful, truthful, and safe for people?
3. Business & Market Factors — Are they practical to deploy at scale?
4. Evaluation Frameworks — What tools and platforms run these evaluations?

Swooping Down: The Details

1. Technical Benchmarks

Knowledge & Reasoning: MMLU tests exam-style accuracy across 57 subjects, while GSM8K focuses on math word problems. GPQA pushes into graduate-level science questions where even PhD experts only get 65% right.

Coding: HumanEval measures whether generated code passes unit tests (using pass@k metrics), but SWE-bench goes further by testing real GitHub bug fixes. LiveCodeBench prevents cheating by using fresh programming contest problems.

👉 These give objective, leaderboard-friendly scores that make models easy to compare.

2. User-Facing Evaluations

Helpfulness: Chatbot Arena lets real users vote in head-to-head comparisons, creating Elo rankings. Human preference studies use expert panels to rate model outputs across multiple dimensions.

Truthfulness: TruthfulQA tests whether models resist common myths and falsehoods. This matters because models can be technically correct but practically misleading.

Harmlessness & Safety: Red-teaming uses adversarial prompts to probe safety limits. The Anthropic HHH guide (Helpful, Honest, Harmless) provides a structured approach. Advanced methods now test for alignment faking — where models pretend to be aligned but aren’t.

👉 These ensure models aren’t just smart, but actually aligned with human needs and values.

3. Business & Market Factors

Cost-effectiveness: API token pricing versus running open models on your own GPUs. Smaller distilled models often provide better value per query than flagship models.

Latency & Throughput: Response time matters for user experience. Queries per second determines how many users you can serve simultaneously.

Energy Efficiency: ChatGPT uses roughly 0.3–0.4 watt-hours per query. Smaller models are significantly greener, which matters for both costs and corporate sustainability goals.

👉 Businesses weigh performance-per-dollar, not just raw accuracy scores.

4. Evaluation Frameworks

Automated Evaluation Platforms: OpenAI Evals and LM Harness make testing easier by providing standardized ways to run hundreds of benchmarks. HELM provides holistic evaluation across multiple dimensions like accuracy, calibration, robustness, and fairness.

Contamination-Resistant Testing: LiveBench addresses the problem where models may have seen benchmark questions during training, making scores meaningless. It updates monthly with fresh content from recent sources, ensuring tests remain valid.

LLM-as-a-Judge: Using GPT-4 or similar models to grade outputs provides a scalable substitute for human raters, achieving 77–85% correlation with human preferences at 98% lower cost. However, judge models have their own biases toward verbose responses and need careful prompt engineering.

👉 These frameworks are the infrastructure that makes large-scale, systematic evaluation possible.

Takeaway

>> Interactive Miro Mind Map <<