Project · Intermediate · about 16 hours
An evaluation harness that blocks bad prompt changes
Turn an LLM ticket-triage prompt into a tested component, with a golden set, code-based graders, a calibrated LLM judge and a CI gate that fails on regressions.
Skills:CI/CD, LLMs and NLP, LLM evaluation, Prompt engineering, Python
Part of: AI/ML Engineer, Forward Deployed Engineer
Data and licences
- PolyAI Banking77 ↗CC BY 4.0
Brief
Your team shipped an LLM that labels incoming banking messages with an intent and an urgency, and drafts a first reply. Every prompt tweak "looks better" in a demo and sometimes breaks something quietly. Build the harness that makes changes safe.
Banking77 gives you real customer queries with trusted intent labels. Urgency and reply quality have no labels, so you will write a rubric and label 50 cases yourself.
What to build
Write the triage function with structured output (a JSON schema or a typed model). Record the model name and prompt version in every result.
Sample 200 test queries stratified by intent. Add 30 edge cases: typos, two intents in one message, prompt injection attempts, messages that are not about banking.
Grade intent and schema validity with code. Grade urgency against your 50 labels.
Write a rubric for the reply draft and an LLM judge that applies it. Measure judge agreement with your own grades on 50 replies before you trust it.
Store a baseline run. On each change, run the suite and fail if intent accuracy drops by more than 2 points or any schema check fails.
Wire it into GitHub Actions, and use a cached or recorded mode so pull requests from forks do not need an API key.
Deliverables
Repository with
triage/,evals/golden.jsonl,evals/run.py,evals/judge.py,.github/workflows/evals.ymlandREPORT.md
How it is graded
Reviewers open a pull request that makes the prompt worse on purpose and check that your CI catches it. They also read how you validated the judge.
Stretch
Track cost and latency per case, and add a cheaper model as a candidate with a side-by-side report.
Deliverables
- A triage function that returns structured output (intent, urgency, reply draft)
- A golden set of at least 200 cases drawn from the test split, plus 30 hand-written edge cases
- Code graders for the structured fields and an LLM judge for reply quality, checked against your own labels
- A CI job that runs the suite on every change to the prompt and fails on a regression
How a reviewer grades it
| Criterion | Weight | What good looks like |
|---|---|---|
| Golden set is versioned, balanced across intents and includes hard and adversarial cases | 25% | |
| Deterministic checks are used wherever possible; the judge is used only for what code cannot check | 20% | |
| Judge agreement with human labels is measured and reported | 25% | |
| CI gate works, with thresholds and a readable report of what changed | 30% |