Fine-Tuning vs Prompting for Intent Classification: A Concrete Comparison
We had 8,000 labeled support intents. I ran the same task through GPT-4o prompting and a fine-tuned distilbert. The answer isn't what the blog posts say.
We had 8,312 labeled examples and a hard latency requirement. I needed to classify customer support messages into 47 intent categories — `billing_dispute`, `account_locked`, `cancel_subscription`, and 44 others — with a p50 under 100ms and a per-inference cost that didn't make our CFO ask questions. The obvious move was to reach for GPT-4o and write a good few-shot prompt. The less obvious move was to spend 47 minutes on a fine-tuning run first.
The results weren't close.
The Setup
**Task:** 47-class intent classification on English support tickets. Labels like `billing_dispute`, `feature_request`, `account_locked`, `cancel_subscription`, etc.
**Training set:** 6,650 examples (80/20 split). Test set: 1,662 examples, stratified.
**Approach A — Prompting GPT-4o:** Few-shot prompt with 10 examples, one per top-level category bucket, plus the full list of 47 intents and descriptions. Temperature 0. Structured output mode for clean JSON.
**Approach B — Fine-tuned distilbert:** `distilbert-base-uncased`, fine-tuning with the Trainer API on the full 6,650 training examples. 5 epochs, lr=2e-5, batch size 32, linear warmup over first 10% of steps. Trained on a single A10G (24GB VRAM) — 47 minutes.
Results
| Approach | Accuracy | F1 (macro) | Latency p50 | Cost per 1k |
|---|---|---|---|---|
| GPT-4o few-shot | 0.847 | 0.823 | 890ms | $2.70 |
| distilbert fine-tuned | **0.921** | **0.899** | **22ms** | $0.004 |
Fine-tuned distilbert wins by 7.4 points accuracy — 40× faster, ~675× cheaper per inference call, and it's not a close race on any dimension that matters for production.
The aggregate hides where each approach actually wins.
Where Prompting Won
On rare intent categories (fewer than 30 training examples), GPT-4o held its ground. For `data_export_request` (n=11 in training), GPT-4o scored 0.73 F1; fine-tuned distilbert scored 0.41. When you have almost no labeled data for a class, the language model's priors are worth more than your fine-tune.
GPT-4o also handled edge cases and ambiguous tickets better. Messages that a human would mark as "actually both `billing_dispute` and `cancel_subscription`" — GPT-4o returned the right primary label more often. distilbert confidently picked wrong.
The Tail-Class Problem Is the Real Argument
The aggregate F1 isn't the number that matters. The per-class breakdown is.
My first Trainer run used the default config — no label smoothing, no class weighting. I assumed: clean dataset, 6,650 examples, this should converge. It did converge, to 0.831 F1 macro, which barely beats GPT-4o's 0.823. I stared at it for twenty minutes, re-ran the eval three times looking for a bug in the metric code. No bug. The training just hadn't done what I expected.
The reason was visible in the per-class numbers, which I hadn't looked at yet. `general_question` has 1,100 training examples. `gdpr_deletion_request` has 19. Without class weights, the model essentially gave up on the tail classes — and `gdpr_deletion_request` is a legally sensitive intent we route to a compliance queue manually. At 0.12 F1 on that class, the fine-tuned model was less reliable than GPT-4o on exactly the cases that mattered most.
Second run: `sklearn.utils.class_weight.compute_class_weight('balanced', ...)`, weights passed to a custom loss in a `Trainer` subclass. F1 macro jumped from 0.831 to 0.899. The table numbers are from that run. The aggregate metric became meaningful only after the class imbalance was addressed.
One other limit worth flagging: distilbert's 512-token ceiling. Support tickets are often longer — signature blocks, quoted email chains, the whole thread pasted in by someone who just wants their invoice resolved. I truncated at 512 with `truncation=True` and lost signal. A model with longer context (`longformer` or similar) is worth testing before committing to this architecture for long-tail support domains.
The Actual Decision
If you're doing classification, the decision mostly comes down to data volume and schema stability. With 500+ labeled examples per class and a stable label set, fine-tune — the accuracy and latency numbers justify it fast. Fewer than 50 labeled examples on important classes means your fine-tuned model will be unreliable on exactly the intents that matter most; prompt or use a hybrid. Hard latency requirements under 100ms rule out API calls entirely. Rapidly changing label schema is the one case where prompting wins on pure economics — retraining cost adds up quickly when you're updating the schema every few weeks.
We went with the fine-tuned distilbert for production. At our volume (roughly 30k tickets/day), GPT-4o would run ~$81/day. distilbert on a single A10G on Lambda Labs ($0.76/hr) runs the full volume with headroom for about $18/day at current pricing.
The same pattern showed up when I benchmarked three embedding models on 50k support tickets: once you have enough labeled data, specialized smaller models often beat the APIs on recall metrics — the domain signal outweighs the general-purpose training.
Three months in
The numbers aren't close. 40× faster, 675× cheaper, 7.4 points better accuracy — if you have 6,000+ labeled examples and a stable label schema, there isn't a version of the analysis where prompting wins. The GPT-4o approach is burning money for worse results at worse latency.
The tail-class problem is real and I don't have a clean answer for it. GPT-4o handled our rare intents better; the fine-tuned model effectively gave up on classes with fewer than 30 training examples. We're running a hybrid now — fine-tuned distilbert as the primary classifier, GPT-4o as a fallback for low-confidence predictions. Whether that's the right tradeoff at our volume is something I'll know better in three months.
The part that almost derailed it was the first bad training run. 0.831 F1, barely beating GPT-4o after two days of setup, close enough to rationalize walking away. If I had, we'd still be on the API at $81/day for a worse classifier. Read the per-class breakdown before you read the aggregate metric — the aggregate will lie to you when your label distribution is skewed, and it's usually skewed.