Evals — LLM & RAG Evaluation

Why evals are different

LLM outputs are open-ended, stochastic, and frequently correct in many surface forms. Exact-match assertions break, and a single bad sample is not a regression. Evals replace assertions with scored aggregates over a dataset, and replace booleans with distributions you can track over time.

• Offline evals — run on a curated dataset in CI or pre-release.
• Online evals — run on a sample of live traffic with user feedback, latency, cost.
• Pairwise / A-B — compare candidate vs baseline; report win-rate.

The eval loop

dataset ──▶ run system ──▶ collect outputs ──▶ score (metric) ──▶ aggregate ──▶ compare to baseline
   ▲                                                                                   │
   └────────────── add failure cases as new examples ◀────────────────────────────────┘

Metric families

RAG evaluation — the triad

RAG failure modes split cleanly into retrieval and generation. The RAG triad (popularized by RAGAS and TruLens) gives you one metric per side plus a relevance check on the query.

Retrieval-only metrics (when you have labeled relevant docs): Recall@k, Precision@k, MRR, nDCG, Hit Rate. Generation-only metrics when ground truth exists: Answer Correctness, Semantic Similarity.

LLM-as-judge

A stronger model scores outputs against a rubric. Cheap to scale, but biased — toward longer answers, its own style, and the first option in a pair. Mitigations: pairwise comparisons with position swap, chain-of-thought rationale, calibrated rubrics, and periodic human spot-checks to validate the judge.

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision

result = evaluate(
    dataset,  # columns: question, answer, contexts, ground_truth
    metrics=[faithfulness, answer_relevancy, context_precision],
)
print(result.to_pandas())

Agent & tool-use evals

For agents, the answer is not enough — the trajectorymatters. Score:

• Task success — did the final state match the goal?
• Tool selection — was the right tool chosen at each step?
• Argument correctness — were arguments well-formed and grounded?
• Trajectory efficiency — steps, tokens, $ vs an optimal path.
• Recovery — did the agent handle a tool error without looping?

Datasets

The dataset is the eval. Curate it deliberately: seed with hand-written examples, mine from production logs, and add every bug report as a permanent test case. Stratify across user intents and difficulty so a single category cannot dominate the aggregate.

• Golden set — small, hand-verified, gates releases.
• Regression set — every bug ever fixed, never removed.
• Synthetic set — generated by an LLM for coverage; treat as lower-trust.
• Adversarial set — jailbreaks, injections, edge inputs.

CI gates & tooling

Wire evals into CI like tests. Block a deploy when the golden set drops more than N%, latency p95 regresses, or cost-per-request spikes. Common stacks:

• LangSmith — datasets, evaluators, experiments, online traces.
• RAGAS — RAG metrics (faithfulness, relevancy, precision/recall).
• TruLens — RAG triad + feedback functions + dashboards.
• DeepEval / promptfoo / Braintrust — pytest-style eval suites and pairwise grading.
• OpenAI Evals / Inspect AI — task framework for model-level benchmarks.

Common pitfalls

• Optimizing a single average — track distributions and per-slice scores.
• Judge model leaks into training of the candidate — use a different family.
• Dataset contamination — keep the golden set out of prompts and logs.
• Reference-based metrics on free-form answers — switch to LLM-as-judge or rubric scoring.
• No human-in-the-loop sample — re-validate the judge against humans monthly.