Fine-tuning

TL;DR

• Fine-tuning continues training a pretrained model on your own dataset, nudging its weights toward your specific task, format, or domain vocabulary.
• Try RAG and good prompting first. Fine-tune only when you need consistent format, domain vocabulary, or cost/latency reduction via a smaller specialized model.
• LoRA often trains well under 1% of parameters and can be competitive with full fine-tuning on many tasks. QLoRA can make 70B-class training fit in some single-GPU configurations, depending on sequence length, batch size, and optimizer settings.
• Data quality is the primary lever. 1K curated examples routinely outperform 100K noisy ones.
• A held-out evaluation set is non-negotiable. Without one, you cannot tell whether fine-tuning helped or just changed the failure mode.
• The economics can be compelling: a fine-tuned 7B model may reduce cost and latency substantially on a narrow task, but quality, throughput, hardware, and operational costs must be measured against the larger model.

30-Second Explanation

Mental model: fine-tuning changes model weights using task-specific examples so a pretrained model behaves more consistently for a target distribution. It can improve format, style, or task behavior, but it does not automatically add missing knowledge or guarantee correctness; compare it with prompting and retrieval on a held-out eval.

The problem it solves

General-purpose models know everything about everything, which means they know nothing specifically about your product. Ask GPT-4 to draft a support reply in your company's voice and you get something competent but generic. It does not know your tone, your internal terminology, or the specific way you acknowledge customer frustration before pivoting to solutions.

You could put all of that context into the system prompt. Sometimes that works. But prompts get long, long prompts cost money at inference, and the model treats every instruction as a temporary hint rather than internalized behavior.

The other driver is cost and latency. A fine-tuned 7B model running on your own hardware may be much cheaper and faster than a larger API model on a narrow task, but the result depends on hardware utilization, traffic shape, and the quality bar. Include training, evaluation, serving, and maintenance in the break-even calculation.

Here is the cost breakdown that makes the case:

That math is hard to argue with. When you need a model that feels like it belongs to your product rather than borrowing time from a general assistant, fine-tuning is the answer.

What is it?

Fine-tuning means continuing the training process on a pretrained base model using your own dataset. The model's weights, which already encode language understanding from pretraining on trillions of tokens, get nudged toward the specific behavior you want.

Think of it like hiring an experienced chef and teaching them your restaurant's recipes. You do not teach them how to cook from scratch. They already know knife skills, heat control, and flavor balance. You just show them your specific dishes, your plating style, and your secret sauce. That is fine-tuning: specialization on top of deep general competence.

There are several distinct approaches under the fine-tuning umbrella: full fine-tuning, LoRA, QLoRA, and instruction fine-tuning. Choosing between them depends on hardware, cost, data, model capacity, and the quality target. The concise definition is: "Fine-tuning adapts a pretrained model's weights to a specific task using domain data." The important follow-up is what prompting, retrieval, and evaluation showed before making that training commitment.

How it works

Full fine-tuning

Full fine-tuning updates every parameter in the model using standard gradient descent on your dataset. For a 7B parameter model, you are touching 7 billion floats. In fp16 precision, that means roughly 14GB just for the weights, plus optimizer states (Adam stores two additional copies), pushing total VRAM to 80GB or more.

This approach has the highest quality ceiling because every weight can shift to fit your task. It also carries the highest risk of catastrophic forgetting, where the model loses general capabilities while over-specializing on narrow data.

The training loop is conceptually simple: take a batch of (input, expected_output) pairs, compute the loss between the model's prediction and the expected output, then backpropagate gradients through every layer. The entire parameter space shifts with each step. This is the same process as pretraining, just on your smaller dataset with a lower learning rate (typically 1e-5 to 5e-5).

Full fine-tuning is often a poor default because of its cost and forgetting risk. Reserve it for cases with sufficient data and compute where adapter methods do not meet the quality, capacity, or serving requirements; model size alone is not a reliable cutoff.

LoRA (Low-Rank Adaptation)

LoRA freezes all base model weights and inserts small trainable matrices into each attention layer. Instead of updating a weight matrix W directly, LoRA adds a low-rank decomposition: delta_W = A x B, where A and B are much smaller matrices. The rank hyperparameter (typically 4 to 64) controls the capacity of these adapters.

The key insight is that the weight updates needed for task adaptation live in a low-dimensional subspace. A full-rank update is wasteful. LoRA typically trains under 0.1% of total parameters, produces results competitive with full fine-tuning on most NLP tasks, and costs roughly 10x less to run.

LoRA adapters are small files (tens of megabytes) that you merge into the base model at serving time or swap between tasks. This makes multi-tenant fine-tuning practical: one base model, many adapters.

# Simplified QLoRA config (using PEFT + bitsandbytes)
from peft import LoraConfig, get_peft_model

lora_config = LoraConfig(
    r=16,                    # Rank: 4-64, higher = more capacity
    lora_alpha=32,           # Scaling factor, typically 2x rank
    target_modules=["q_proj", "v_proj"],  # Which layers get adapters
    lora_dropout=0.05,
    task_type="CAUSAL_LM",
)

model = get_peft_model(base_model, lora_config)
print(f"Trainable params: {model.num_parameters(only_trainable=True):,}")
# Output: Trainable params: 4,194,304  (vs 7B total)

QLoRA (Quantized LoRA)

QLoRA combines 4-bit NF4 quantization of the base model with LoRA adapters trained in fp16. The base weights are compressed to 4-bit precision (cutting VRAM by roughly 4x), while the adapter computations remain in full precision to preserve gradient quality.

In some configurations this can fit a 70B model onto a single 40GB A100 GPU for training, but feasibility depends heavily on sequence length, batch size, checkpointing, and optimizer settings. QLoRA is a useful starting point for teams with limited GPU memory; compare it with LoRA and managed fine-tuning on the actual workload.

The quality trade-off is smaller than you would expect. The QLoRA paper showed results within 1-2% of full fine-tuning on most benchmarks, while using a fraction of the compute.

Comparing the approaches

Instruction fine-tuning

Instruction fine-tuning trains a base model on (instruction, response) pairs so it learns to follow directions. This is how raw GPT-3 became InstructGPT, and how Llama 2 Base became Llama 2 Chat. Without instruction tuning, base models autocomplete text rather than answering questions.

The data format is critical. For single-turn tasks, use prompt-completion pairs. For chat models, use the multi-turn format with system/user/assistant roles matching the model's expected chat template. Mismatched formats are the most common source of poor results that do not obviously look like overfitting.

If someone asks how a base model becomes a chat model, explain that instruction tuning and preference or reinforcement-based alignment are common stages, but not every model uses the same recipe.

Data quality and curation