A support reply assistant with structured output

#Brief

A support team wants first-draft replies so agents spend time checking rather than typing. Before any agent sees a draft, you need to know how often the assistant gets the category right and when it should hand over to a human.

The Bitext messages come with category and intent labels, and placeholders such as {{Order Number}} in place of personal details, which is how you should treat customer data too.

#What to build

1. Sample 100 messages stratified by category as a held-out test set, and 50 more as a development set.

2. Write a prompt with definitions for the 11 categories, rules for when a human is needed (legal threats, payment disputes, anything not about the store) and examples for confusions you find on the development set.

3. Call the model with structured output (a JSON schema or typed model) and validate every response.

4. Score category and intent accuracy on the test set, once. Grade 20 reply drafts yourself with a three-point rubric.

5. Add ten off-topic or adversarial messages of your own and check that they are handed to a human.

#Deliverables

• Repository with prompts/, assistant.py, eval/, results.md and README.md

#How it is graded

Reviewers rerun the evaluation from saved outputs and read your failure cases. Getting the process right counts more than the score.

#Stretch

Compare two models on the same set and recommend one on accuracy, cost and latency.