Project · Advanced · about 32 hours
Run a customer pilot end to end
Take a simulated customer from discovery to a decision: scope a pilot, build a cited support assistant over their policies, evaluate it, demo it on their data and hand it over.
Skills:Customer-facing delivery, LLMs and NLP, LLM evaluation, Monitoring and observability, Retrieval-augmented generation (RAG), Stakeholder management
Part of: Forward Deployed Engineer
Data and licences
- Bitext customer support dataset (synthetic, 26,872 messages) ↗CDLA-Sharing-1.0
Brief
This is the forward deployed job in one project. A fictional online store (you write its policies: returns, refunds, delivery times, payment methods, account deletion) wants an assistant that drafts policy-correct replies to customer messages. The Bitext messages stand in for their traffic.
Work in stages, and do not start a stage before the previous one is written down.
What to build
Discovery and plan. Role-play the discovery call, then write the pilot plan: scope (which intents), success criteria, guardrails and a decision date.
Policies. Write 10 to 15 short policy documents. Include two that conflict and one that changed recently, because real customers have both.
Assistant. Retrieve relevant policy sections, draft a reply with citations, and hand over to a human for disputes, legal threats and anything not covered.
Evaluation. Build a set of at least 150 messages across the chosen intents plus hand-written edge cases. Check citations with code and reply correctness with a validated judge. Compare against the success criteria.
Demo and decision. Write the demo script, record it, and write the readout with a recommendation: roll out, extend or stop.
Handover. Architecture page, runbook, monitoring dashboard and alert thresholds, and one field note for your product team.
Deliverables
Repository with
01-discovery/,02-policies/,03-assistant/,04-evals/,05-demo/,06-handover/
How it is graded
Reviewers play the customer's sponsor and IT lead. They read the readout, watch the demo, ask five questions and try to operate the system from your runbook alone.
Stretch
Update one policy after "go-live" and show the change flowing through ingestion, evaluation and a release note to the customer.
Deliverables
- Discovery notes, a pilot plan with success criteria and a baseline
- A support assistant that answers from a policy document set you write, with citations and hand-over rules
- An evaluation suite and a readout against the success criteria
- A demo script and recording, a handover pack (architecture, runbook, monitoring) and one field note for the product team
How a reviewer grades it
| Criterion | Weight | What good looks like |
|---|---|---|
| Pilot plan with measurable criteria agreed before building | 20% | |
| Assistant is grounded in the policies, cites them and hands over when it should | 25% | |
| Evaluation and readout are honest, including where the pilot fell short | 25% | |
| Demo, handover pack and field note are professional and usable by someone else | 30% |