Fine-tuning has become a reflex. The model does not behave exactly how you want, so the next thought is: let's fine-tune it. In most cases this is the wrong decision, taken too early, for the wrong reasons.
What Fine-Tuning Actually Changes
Fine-tuning adjusts the weights of a pre-trained model using your data. It is useful for two things:
- ▸Teaching the model a new skill it genuinely lacks — a task it has never seen examples of
- ▸Changing the model's default style or format — when you need consistent output structure across thousands of calls without spelling it out in every prompt
That is a narrow list. Most application requirements do not fall into either category.
Fine-tuning does not reliably:
- ▸Inject new factual knowledge (use RAG for this)
- ▸Prevent hallucination (it can actually make it worse on out-of-domain queries)
- ▸Improve reasoning on complex tasks (larger models or chain-of-thought prompting do this better)
- ▸Fix problems caused by bad prompts
The Real Cost
Teams underestimate what fine-tuning actually costs.
Data preparation: You need 100-1000+ high-quality labeled examples. Collecting, cleaning, and formatting this data takes more time than the training itself. Bad training data produces a confidently wrong model.
Training cost: Fine-tuning GPT-3.5 on 1000 examples costs ~$8-15. Fine-tuning a 7B parameter open-source model on a GPU (A100) for 3 hours costs ~$15-20. That is not the expensive part.
Evaluation cost: How do you know your fine-tuned model is better? You need an evaluation set, a baseline, and a repeatable benchmark. Building this properly takes days.
Maintenance cost: When the base model is updated, your fine-tuned version is not. You either stay on an older model or re-fine-tune. This is an ongoing cost most teams do not plan for.
Iteration cost: If your first fine-tuned model is not good enough, you need more data or a different approach. Each iteration restarts the cycle.
What to Try Before Fine-Tuning
Better system prompts:
Most "the model doesn't follow my format" problems are solved by a more explicit system prompt with examples.
system = """You are a customer support agent for HackerSchool. ALWAYS respond in this exact JSON format: { "answer": "your answer here", "confidence": "high|medium|low", "escalate": true|false, "suggested_article": "article title or null" } Never add explanation outside the JSON. Never omit any field."""
If the model consistently follows this format across 100 test cases, you do not need fine-tuning.
Few-shot examples in the prompt:
3-5 examples of input/output pairs in the prompt can shift model behaviour dramatically.
prompt = """Classify customer messages as: billing, technical, general. Examples: Message: "I was charged twice this month" → billing Message: "The video won't load on my laptop" → technical Message: "What courses do you offer?" → general Message: "{user_message}" →"""
RAG for knowledge:
If the model gives wrong answers because it lacks up-to-date or domain-specific knowledge, fine-tuning will not fix this — it memorises patterns, not facts reliably. RAG retrieves the right information at inference time.
Larger model:
If you are using GPT-3.5 or a 7B parameter model and getting poor results on complex reasoning, switching to GPT-4o or a 70B model often outperforms a fine-tuned smaller model at lower engineering cost.
When Fine-Tuning Is Actually the Right Call
There are legitimate cases:
- ▸Consistent output format at scale: You make 10 million API calls per month. Putting a 2000-token system prompt on every call costs $400/month. Fine-tuning the format into the model eliminates most of that prompt, saving real money.
- ▸Proprietary style or voice: You need the model to write in a very specific brand voice that cannot be described in a prompt — it has to be demonstrated through examples.
- ▸Latency-sensitive paths: A fine-tuned 7B model can outperform a prompted 70B model on a specific task at 10x lower latency and cost.
- ▸Edge task generalisation: The task is genuinely not in the pre-trained model's distribution — a specific medical coding format, a proprietary query language, a niche classification scheme.
The Decision Test
Before fine-tuning, run this test:
- ▸Write the best prompt you can for the task
- ▸Add 5 few-shot examples
- ▸Test on 50 real inputs and measure accuracy
If accuracy is acceptable: you are done. Prompt engineering worked.
If accuracy is not acceptable: look at the failures. Are they format issues, reasoning failures, or knowledge gaps? Format → better prompt. Reasoning → larger model. Knowledge → RAG.
If none of those help and you have 500+ labeled examples: now consider fine-tuning.
The Bottom Line
Fine-tuning is a powerful tool used too early by most teams. It solves a narrow set of problems and creates an ongoing maintenance burden. Prompts are cheap, fast to iterate, and transparent — you can read them. Fine-tuned weights are a black box.
Exhaust prompt engineering first. The answer is usually there.