An evaluation harness for an LLM feature

5 min read

Why it matters

LLM evaluation appeared in about a third of the Forward Deployed Engineer job descriptions we read. A prompt that looks better on three examples can be worse on the fifty that matter. An evaluation harness turns "it seems better" into "category accuracy went from 0.84 to 0.91 and nothing else dropped", which is what lets a team ship with confidence.

The concept

Cases: realistic inputs with the correct outputs, stored as data (JSON lines). Start with 20 to 50 from real traffic, including hard and edge cases; add every production failure you find.
Checks: exact match for structured fields (category, urgency), programmatic checks for rules (did it escalate a legal threat?), and a model grader for open-ended text (next lesson).
Metrics per field, not a single score, plus an error table showing which mistakes happen.
Versioned prompts in files, so each result is tied to the exact prompt that produced it.
Run the eval on every change: prompt, model, effort level or retrieval.

Worked example: evaluating support triage

The feature is the triage function from Calling an LLM from production code, with its system prompt moved to a versioned file:

prompts/triage-v2.md

You triage customer messages for an Indian online grocery service.

<categories>
billing: charges, double payments, invoices, UPI or card issues
delivery: late, missing or damaged deliveries, wrong items
refund: the customer explicitly asks for money back
account: login, address, profile, subscription settings
other: anything else
</categories>

<urgency>
high: money taken without delivery, safety issues, or the customer says it is urgent today
medium: a problem with an order in progress
low: questions, feedback, and requests without a deadline
</urgency>

Set needs_human to true for legal threats, safety issues or payments over 10,000 rupees.
Write the summary as one sentence in English, stating only what the message says.

Labelled cases, one per line (a real set needs many more):

triage_eval.jsonl

{"id": 1, "message": "Order 4471 was charged twice on my UPI and the milk never came. Please fix today!", "category": "billing", "urgency": "high", "needs_human": false}
{"id": 2, "message": "The eggs in my order were broken. Can I get a replacement tomorrow?", "category": "delivery", "urgency": "medium", "needs_human": false}
{"id": 3, "message": "I want my money back for the rotten mangoes, 340 rupees.", "category": "refund", "urgency": "medium", "needs_human": false}
{"id": 4, "message": "How do I change my delivery address for future orders?", "category": "account", "urgency": "low", "needs_human": false}
{"id": 5, "message": "Your delivery person was rude and nearly hit my child with his bike. I will go to the police.", "category": "other", "urgency": "high", "needs_human": true}
{"id": 6, "message": "Bulk order for our office canteen, 18,500 rupees debited but not confirmed.", "category": "billing", "urgency": "high", "needs_human": true}

The harness:

eval_triage.py

"""Evaluate the support triage prompt on a labelled set: per-field accuracy, confusion, and a judged summary score."""
import json
from collections import Counter
from pathlib import Path
from typing import Literal

import anthropic
from pydantic import BaseModel

client = anthropic.Anthropic()
PROMPT_VERSION = "triage-v2"
SYSTEM = Path(f"prompts/{PROMPT_VERSION}.md").read_text()


class Triage(BaseModel):
    category: Literal["billing", "delivery", "refund", "account", "other"]
    urgency: Literal["low", "medium", "high"]
    summary: str
    needs_human: bool


class Judgement(BaseModel):
    faithful: bool          # says nothing the message does not support
    complete: bool          # captures the customer's main problem
    reason: str


def triage(message: str) -> Triage:
    response = client.messages.parse(model="claude-opus-5-5", max_tokens=1024, system=SYSTEM,
                                     output_config={"effort": "low"},
                                     messages=[{"role": "user", "content": message}], output_format=Triage)
    return response.parsed_output


def judge(message: str, summary: str) -> Judgement:
    response = client.messages.parse(
        model="claude-sonnet-5-5", max_tokens=1024,          # a cheaper model as the grader
        system="You grade one-sentence summaries of customer messages. Be strict and literal.",
        messages=[{"role": "user", "content": f"<message>{message}</message>\n<summary>{summary}</summary>\n"
                   "Is the summary faithful (no invented facts) and complete (captures the main problem)?"}],
        output_format=Judgement)
    return response.parsed_output


cases = [json.loads(line) for line in open("triage_eval.jsonl")]
results, confusion = [], Counter()
for case in cases:
    got = triage(case["message"])
    verdict = judge(case["message"], got.summary)
    confusion[(case["category"], got.category)] += 1
    results.append({"id": case["id"], "category_ok": got.category == case["category"],
                    "urgency_ok": got.urgency == case["urgency"], "human_ok": got.needs_human == case["needs_human"],
                    "summary_ok": verdict.faithful and verdict.complete, "got": got.model_dump()})

n = len(results)
report = {field: round(sum(r[field] for r in results) / n, 3) for field in ("category_ok", "urgency_ok", "human_ok", "summary_ok")}
report.update(prompt=PROMPT_VERSION, cases=n,
              category_errors={f"{want}->{got}": c for (want, got), c in confusion.items() if want != got})
print(json.dumps(report, indent=1))
Path(f"results-{PROMPT_VERSION}.jsonl").write_text("\n".join(json.dumps(r) for r in results))

The report has one accuracy per field and a list such as {"refund->billing": 2} showing exactly which confusions happen, and every per-case result is saved next to the prompt version. Read the failures, change one thing in the prompt (for example, add an example of a refund request that mentions a charge), save it as triage-v3.md, rerun, and compare field by field.

Case 6 is deliberately tricky: the payment is over 10,000 rupees, so needs_human must be true even though the tone is calm. Such rule cases catch prompts that rely on tone.

Common mistakes

A handful of easy cases. Include edge cases, ambiguous messages, other languages (Hinglish), and past failures.
One overall score. Averages hide a field that got worse. Report per field and per case type.
Changing several things at once. You will not know which change helped.
Evaluating on the examples inside the prompt. They will pass by construction; keep them separate.

Interview angle

"How do you know your LLM feature works?" Describe the labelled set and where it came from, per-field metrics, error analysis, model graders for free text with human calibration, and running it on every change. Mention that production failures become new cases.

Practice task

Write 30 more cases, including five in Hinglish and five that should be other.
Run the harness with effort set to low and to medium, and compare accuracy and cost per case.
Add a check that summary never contains a phone number or an order total that is not in the message.

#Why it matters

#The concept

#Worked example: evaluating support triage

#Common mistakes

#Interview angle

#Practice task

#Further reading