SkillAI/ML EngineerForward Deployed Engineer
Library · practice
LLM evaluation
Measure whether an LLM feature works, with labelled test sets, automatic checks and calibrated model graders, and stop regressions before release.
LLM evaluation appeared in 31% of the Forward Deployed Engineer and 20% of the AI/ML Engineer postings we read, and it is the practice that separates demos from products.
The harness in these lessons evaluates the support triage feature from the LLMs shelf. The metric code runs offline; calling the model needs an API key.
Beginner lessons for LLM evaluation are being written.
- 1IntermediateA cited question-answering assistant over the Kubernetes docs
- 2IntermediateAn evaluation harness that blocks bad prompt changes
- 3AdvancedA data question agent with guardrails and an eval suite
- 4AdvancedFine-tuning versus prompting, decided with numbers
- 5AdvancedRun a customer pilot end to end