Y

LLM evaluation · Advanced

Trust, but verify, your model grader

3 min read

Why it matters

Summaries, explanations and answers cannot be checked by exact match, so teams use a model as a grader ("LLM-as-judge"). A grader that disagrees with people quietly steers the product in the wrong direction. Senior AI engineers calibrate graders against human labels and wire evaluations into CI so regressions never ship.

The concept

  • A judge is a prompt and model that returns a structured verdict, ideally binary or on a small scale, with a reason.

  • Calibrate it: have people label 50 to 100 outputs, run the judge on the same outputs, and measure agreement.

  • Raw agreement can look high just because most outputs are good. Cohen's kappa corrects for agreement by chance: around 0.6 to 0.8 is substantial, above 0.8 strong.

  • Look at the disagreements: they show where the rubric is vague.

  • Regression gates: in CI, fail the build if a key metric drops below a threshold or below the current production version by more than a margin.

Worked example: judge versus people on 20 summaries

Suppose two reviewers agreed labels (1 = good summary, 0 = bad) for 20 outputs, and the judge from An evaluation harness for an LLM feature graded the same 20:

calibrate.py
from sklearn.metrics import cohen_kappa_score, confusion_matrix

human = [1, 1, 0, 1, 0, 1, 1, 0, 1, 1, 0, 1, 1, 1, 0, 1, 1, 0, 1, 1]
judge = [1, 1, 0, 1, 1, 1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 1, 1, 1, 1, 1]

agreement = sum(h == j for h, j in zip(human, judge)) / len(human)
print("agreement", agreement, "kappa", round(cohen_kappa_score(human, judge), 3))
print(confusion_matrix(human, judge, labels=[1, 0]))
output
agreement 0.85 kappa 0.625
[[13  1]
 [ 2  4]]

85% agreement sounds good, but kappa is 0.625: substantial, not strong. The confusion matrix shows the pattern: the judge passed two summaries people rejected (bottom left) and failed one they accepted. Read those three cases. Typically the rubric is ambiguous ("complete" means what, exactly?) or the judge is lenient about small invented details. Tighten the rubric with examples of each failure, rerun, and aim for kappa above 0.7 before relying on the judge.

Then gate releases. A simple CI check against the stored results:

gate.py
"""Fail CI if the candidate prompt is worse than production on any key field."""
import json
import sys

THRESHOLDS = {"category_ok": 0.90, "human_ok": 0.95}      # absolute floors
MAX_DROP = 0.02                                          # allowed drop versus production

production = json.load(open("report-production.json"))
candidate = json.load(open("report-candidate.json"))

failures = []
for field, floor in THRESHOLDS.items():
    if candidate[field] < floor:
        failures.append(f"{field} {candidate[field]} is below the floor {floor}")
    if candidate[field] < production[field] - MAX_DROP:
        failures.append(f"{field} dropped from {production[field]} to {candidate[field]}")

print("\n".join(failures) or "all gates passed")
sys.exit(1 if failures else 0)

needs_human gets the higher floor because a missed escalation (a legal threat handled by a bot) costs far more than a wrong category.

Common mistakes

  • Trusting a judge you never calibrated. Measure agreement first.

  • Using the same model and prompt to generate and to grade without checks. Self-preference is a known bias; use a different prompt, a rubric and calibration.

  • 1 to 10 scores. Judges and people disagree more on fine scales. Prefer binary checks with reasons.

  • Gates on the average only. Protect the fields with the highest cost of error separately.

Interview angle

"How would you evaluate a summarisation feature?" Describe a rubric, a model judge with structured output, calibration against human labels with agreement and kappa, reading disagreements, and CI gates with per-field floors. Mention cost: judges run on a cheaper model where calibration allows.

Practice task

  1. Label 30 real outputs yourself, run the judge and compute kappa.

  2. Rewrite the rubric to fix the most common disagreement and measure again.

  3. Add gate.py to a GitHub Actions workflow that runs on pull requests touching prompts/.

Further reading