Fine-Tuning vs Prompt Engineering
Choosing how to adapt an LLM — the ladder from prompting to few-shot to RAG to fine-tuning, the facts-vs-behavior rule, and parameter-efficient LoRA/QLoRA.
There are several ways to make an LLM do what you want, from prompt engineering and few-shot examples to RAG and fine-tuning. The art is choosing the cheapest approach that works: most needs are met by prompting or RAG. The key rule — RAG for new facts, fine-tuning for new behavior — plus parameter-efficient methods like LoRA/QLoRA, lets you adapt models effectively without unnecessary cost.
There's a spectrum of ways to make an LLM do what you want, from cheapest to most involved: prompt engineering, few-shot examples, RAG, and fine-tuning. The discipline is to climb the ladder only as far as you must — most problems are solved by a better prompt or RAG, not by training.
Prompt engineering shapes behavior purely through instructions — instant, free, and endlessly iterable. Few-shot adds a handful of examples in the prompt to demonstrate the desired pattern or format. Together they handle a surprising share of use cases with no training and no infrastructure.
The key decision rule: use RAG to give the model new knowledge (facts, private/current data) and fine-tuning to give it new behavior (a consistent style, output format, tone, or specialized skill). Fine-tuning bakes patterns into the weights; it does not reliably teach new facts and can't keep them current.
Full fine-tuning updates every weight — powerful but expensive and memory-hungry, with a risk of catastrophic forgetting. LoRA freezes the base model and trains small low-rank adapter matrices (~0.1–1% of parameters), capturing most of the benefit cheaply. QLoRA adds 4-bit quantization so you can fine-tune large models on a single GPU, and adapters are tiny and swappable.
Key Concepts
Steering the model via instructions alone — no training, instant iteration, lowest cost.
Putting a few examples in the prompt to demonstrate the desired pattern or format.
Updating model weights on labeled examples to bake in behavior, style, or a skill.
Parameter-efficient fine-tuning that freezes the base and trains tiny low-rank adapters (~0.2% of params).
LoRA on a 4-bit quantized base model — fine-tune large models on a single consumer GPU.
RAG for new FACTS (knowledge), fine-tuning for new BEHAVIOR (style/format/skill). Often combine both.
1# Climb the ladder of adaptation — cheapest first.23# 1) Prompt engineering — change instructions, zero training:4system = "You are a terse SQL expert. Reply with ONLY the query, no prose."56# 2) Few-shot — teach a pattern with examples in the prompt:7examples = [("active users last 7d", "SELECT ... WHERE last_seen > now()-'7d'"), ...]89# 3) RAG — inject KNOWLEDGE (facts the model wasn't trained on) at query time.1011# 4) Fine-tune — change the WEIGHTS to bake in behavior/style/skill.12# Prefer LoRA/QLoRA (PEFT): freeze the base, train tiny adapters.13from peft import LoraConfig, get_peft_model14config = LoraConfig(r=8, lora_alpha=16, target_modules=["q_proj","v_proj"],15 lora_dropout=0.05, task_type="CAUSAL_LM")16model = get_peft_model(base_model, config) # ~0.2% of params are trainable17# QLoRA = load the base in 4-bit, then LoRA → fine-tune a 7B+ model on one GPU.1819# RULE: RAG for new FACTS, fine-tuning for new BEHAVIOR/format/style.
Knowing when to prompt, retrieve, or fine-tune is one of the most practical and cost-impactful decisions in AI engineering — and a frequent interview question. The facts-vs-behavior heuristic and parameter-efficient fine-tuning (LoRA/QLoRA) are essential modern knowledge for building capable AI features without burning budget on the wrong approach.
Common Pitfalls
1Fine-Tuning to Fix a Knowledge Gap (the Wrong Tool)
A team wants the model to answer questions about their constantly-changing product catalog, so they fine-tune it on the catalog. Answers are still wrong and stale, and every catalog update would require retraining.
They used fine-tuning to inject facts. Fine-tuning teaches behavior/patterns, not reliable, updatable knowledge — the model blurs and hallucinates facts, and frozen weights can't track a changing catalog.
Switch to RAG: index the catalog in a vector store and retrieve the relevant entries at query time. Knowledge stays current (just re-index), answers are grounded and citable, and there's no retraining. Reserve fine-tuning for behavior, like enforcing a strict response format.
Takeaway: Match the tool to the need: RAG for facts, fine-tuning for behavior. Trying to fine-tune knowledge into a model is expensive, stale, and hallucination-prone — RAG is the right answer for current, factual data.
2Baking in a Consistent Output Format Cheaply
An app needs the model to always return responses in a very specific structured style. Prompt instructions get it right most of the time, but it occasionally drifts — and the long formatting instructions inflate every prompt's token cost.
Behavior that must be reliable and is hard to fully specify in a prompt is a poor fit for prompting alone; the long instructions are costly and still imperfect, and full fine-tuning seems too expensive.
Fine-tune with LoRA on a few hundred examples of the desired format. The adapter bakes the behavior into the model, so outputs are consistent without verbose prompt instructions — and LoRA/QLoRA makes the training cheap and the adapter tiny and swappable.
Takeaway: When a behavior must be reliable and is hard to fully prompt, LoRA fine-tuning bakes it in cheaply. Parameter-efficient methods make fine-tuning a practical, low-cost option — not a last resort.
Join the discussion
Comments
Loading comments...
Cookie preferences
We use essential cookies for sign-in and preferences. With your permission, we also use basic analytics to improve lessons.