LLMs and NLP · Advanced
Prompting, retrieval or fine-tuning?
Why it matters
Fine-tuning appeared in 28% of the AI/ML Engineer job descriptions we read, and "RAG" in 30%. Teams often fine-tune when a better prompt would do, or build retrieval when the problem is missing labels. Senior AI engineers are expected to choose with evidence: the cheapest approach that clears the quality bar and that the team can maintain.
The concept
Adapt a model in this order, and stop when the bar is met:
Prompting. Clear instructions, a few examples, structured output. Minutes to change, no training data needed. Fails when the model lacks knowledge or when behaviour must be extremely consistent at high volume.
Retrieval (RAG). Give the model the right documents at question time. Best when the gap is knowledge that changes often (policies, product catalogues, tickets). See the RAG shelf.
A small task-specific model. Train a classifier or extractor on labelled examples, possibly on top of embeddings. Cheap and fast at high volume; needs labels.
Fine-tuning a language model. Adjust weights for style, format or a narrow skill. Needs hundreds to thousands of good examples, an evaluation pipeline and a plan for retraining.
Questions that decide it: is the gap knowledge or behaviour? How often does it change? How many labelled examples exist? What are the volume, latency and cost limits?
Worked example: how many labels do you need?
Before deciding to label thousands of examples, measure the learning curve. Using the SMS spam data from Text classification with TF-IDF and embeddings, train an embedding classifier on a balanced sample of n spam and n legitimate messages:
import pandas as pd
from sentence_transformers import SentenceTransformer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import f1_score
from sklearn.model_selection import train_test_split
sms = pd.read_csv("SMSSpamCollection", sep="\t", header=None, names=["label", "text"])
train, test = train_test_split(sms, test_size=0.25, random_state=42, stratify=sms.label)
encoder = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
test_vectors = encoder.encode(test.text.tolist(), batch_size=64, normalize_embeddings=True)
for n in (10, 50, 200):
few = pd.concat([
train[train.label == "spam"].sample(n, random_state=1),
train[train.label == "ham"].sample(n, random_state=1),
])
model = LogisticRegression(max_iter=2000).fit(
encoder.encode(few.text.tolist(), normalize_embeddings=True), few.label)
print(n, "per class: spam F1 %.3f" % f1_score(test.label, model.predict(test_vectors), pos_label="spam"))| examples per class | spam F1 |
|---|---|
| 10 | 0.644 |
| 50 | 0.786 |
| 200 | 0.833 |
Two lessons from 30 lines of code:
Quality climbs steeply at first and then flattens. That curve, not a guess, tells you whether labelling another 1,000 examples is worth it.
The full TF-IDF model trained on about 4,200 messages still scores 0.946. For this task, more labels plus a simple model beat a small labelled set on a general embedding.
The same experiment design works for comparing a prompted LLM (zero-shot, then few-shot) against trained models: one fixed test set, one metric, several approaches, and a table like this one. Put cost per 1,000 predictions in the table too.
Common mistakes
Fine-tuning to add knowledge. Facts change; retrieval updates instantly, weights do not.
No fixed test set. Without it, every approach "looks good" in a notebook.
Ignoring maintenance. A fine-tuned model must be retrained when the base model or the data changes. Budget for it.
Comparing on quality alone. Latency, cost per call and privacy (can the data leave your servers?) often decide.
Interview angle
"When would you fine-tune instead of using RAG?" Answer: fine-tune for behaviour (format, tone, a narrow skill) with enough good examples and a stable task; use retrieval for knowledge that changes; try prompting first. Then describe how you would prove the choice with a fixed evaluation set and a learning curve.
Practice task
Extend the learning curve to 5, 20, 100 and 400 examples per class and plot it.
Sample the few-shot set with the real class ratio instead of balanced. What happens to precision?
Write a one-page decision record for a real use case: options, evidence, decision, and when to revisit.