Y

Project · Advanced · about 24 hours

Fine-tuning versus prompting, decided with numbers

Solve the same 77-intent task three ways (prompting an LLM, embeddings plus a classifier, fine-tuning a small transformer) and recommend one on accuracy, cost, latency and maintenance.

Skills:Deep learning, LLMs and NLP, LLM evaluation, Machine learning

Part of: AI/ML Engineer, Forward Deployed Engineer

Data and licences

Brief

"Should we fine-tune?" is one of the most common questions an AI/ML engineer is asked. Answer it with an experiment instead of an opinion.

For reference, on Banking77's 3,080 test queries, logistic regression on TF-IDF features scores about 89% accuracy and logistic regression on all-MiniLM-L6-v2 embeddings about 93%. Your fine-tuned and prompted systems need to beat these to earn their cost.

What to build

  1. Fix one test sample of 1,000 queries stratified by intent. Everything is scored on it.

  2. Prompting. Give an LLM the 77 intent names with one example each and ask for structured output. Fix the prompt on a validation sample before you touch the test sample.

  3. Embeddings plus classifier. Reproduce the reference baseline.

  4. Fine-tuning. Fine-tune a small encoder (for example a MiniLM or DistilBERT checkpoint) for classification. Plot accuracy against training set size.

  5. Measure latency per message and estimate cost per 1,000 messages for each system, including training time. Write down the prices you assumed and the date.

  6. Note what happens when a 78th intent is added: what does each system need?

Deliverables

  • Repository with prompting/, embeddings/, finetune/, results.csv and RECOMMENDATION.md

How it is graded

Reviewers rerun the scoring on your saved predictions and read the recommendation. The best answers say which system to pick and exactly when that choice would flip.

Stretch

Try parameter-efficient fine-tuning (LoRA) on a small decoder model and add it to the comparison.

Deliverables

  • Three systems evaluated on the same held-out test sample
  • A learning curve for the fine-tuned model with 10, 25, 50 and 100% of the training data
  • A cost and latency table per 1,000 messages, with the assumptions written down
  • A one-page recommendation for a team that must pick one

How a reviewer grades it

CriterionWeightWhat good looks like
Fair comparison (same test set, no test tuning, same label set, prompts fixed before testing)30%
Fine-tuning done correctly with a validation set and a saved best checkpoint25%
Cost, latency and maintenance compared with stated assumptions20%
Recommendation follows from the evidence and names when it would change25%