Run a fine-tuning job

Start with the base model

Fine-tuning starts from a base or instruction-tuned model. Pick the smallest model that can already do the task reasonably with a strong prompt. If the base model cannot follow the instruction at all, fine-tuning may not rescue it. If a smaller model can do the job after tuning, it may be cheaper and faster to serve than a larger general model.

The base model is part of the experiment record. A dataset tuned on one base may not behave the same on another. If you switch the base model, rerun the evals and treat it as a new model line.

Two paths: managed API vs self-serve

Most teams choose between a managed fine-tuning API (OpenAI, Anthropic, Google, and similar) and a self-serve GPU job (Hugging Face TRL, Axolotl, Unsloth). The loop is the same: upload or point to data, configure hyperparameters, launch, monitor loss, eval checkpoints, ship the winner. The difference is who owns GPUs, base model access, and serving.

Walkthrough: OpenAI fine-tuning API

This path fits when your product already runs on a provider's chat models and your dataset is in messages JSONL (lesson 02). Assistant-only loss masking is handled by the platform.

1. Prepare JSONL. One JSON object per line with a messages array. Validate schema and row count before upload.
2. Upload the file. Use the Files API with purpose fine-tune, or upload in the dashboard.
3. Create a job. Specify base model (e.g. gpt-4o-mini-2024-07-18), training file ID, and optional validation file. Set hyperparameters if the API exposes them (epochs, batch size, learning rate multiplier).
4. Monitor events. Poll job status or subscribe to webhooks. Watch for failures (format errors, token limits) and training metrics when exposed.
5. Eval before swap. Run your golden set against the new ft:... model ID. Compare to base model + best prompt on the same eval.
6. Deploy. Point production requests at the fine-tuned model name. Keep the previous model ID for rollback.

# Sketch: create job (OpenAI Python SDK)
client.fine_tuning.jobs.create(
  training_file="file-abc123",
  model="gpt-4o-mini-2024-07-18",
  validation_file="file-def456",  # optional
  hyperparameters={"n_epochs": 2},
)

Walkthrough: LoRA on Llama with Axolotl or TRL

This path fits open models on your own GPU. Axolotl uses a YAML config; TRL uses a Python script. Both rely on PEFT LoRA and assistant-only masking via the model's chat template.

1. Install stack. CUDA-compatible PyTorch, transformers, peft, trl, datasets. For Axolotl, follow its install guide; for Unsloth, its patched install.
2. Point at dataset. JSONL or Hugging Face dataset with a messages column. Axolotl: set datasets and dataset_prepared_path in YAML. TRL: load with load_dataset("json", data_files=...).
3. Configure LoRA. Base model (e.g. meta-llama/Llama-3.1-8B-Instruct), load_in_4bit: true for QLoRA, rank/alpha from lesson 03, sequence_len covering your longest example.
4. Launch training. Axolotl: accelerate launch -m axolotl.cli.train config.yml. TRL: SFTTrainer with training_args and PEFT config.
5. Monitor. Watch training and eval loss in WandB, TensorBoard, or logs. Save checkpoints every N steps.
6. Eval checkpoints. Run inference with each adapter on your held-out set. Pick the checkpoint that wins task evals, not just lowest loss.
7. Merge or serve adapter. Merge for a single artifact, or load adapter at serve time (lesson 06).

# Axolotl YAML excerpt (conceptual)
base_model: meta-llama/Llama-3.1-8B-Instruct
load_in_4bit: true
adapter: lora
lora_r: 16
lora_alpha: 32
lora_target_modules: [q_proj, k_proj, v_proj, o_proj]
datasets:
  - path: data/train.jsonl
    type: chat_template
val_set_size: 0.1
num_epochs: 2
learning_rate: 2e-4
sequence_len: 4096

Hyperparameter starting points

Treat this table as a first run, not a guarantee. Your dataset size and base model may need adjustment after one experiment.

The knobs that matter

The main hyperparameters are familiar from training: learning rate, batch size, number of epochs, sequence length, and the adapter settings if you use LoRA. You do not need to memorize every option. You need to understand what each failure looks like.

• Learning rate too high: the model changes too aggressively. Loss may spike, outputs become unstable, and the tune can erase useful behavior.
• Learning rate too low: the run barely moves. Loss falls slowly and task behavior does not change enough.
• Too many epochs: the model memorizes the training set and gets worse on held-out data.
• Sequence length too short: examples get truncated and the model learns from broken conversations.
• Batch size too small: updates are noisy. This can work, but the run may need a lower learning rate or gradient accumulation.

Watch validation, not just training

Training loss should usually go down. That only proves the model is fitting examples it sees. Validation loss measures the same objective on held-out examples. If training loss keeps falling while validation loss rises, the model is overfitting. It is getting better at the training set and worse at generalizing.

Still, validation loss is not enough. A model can have a slightly worse loss but better product behavior if it produces valid JSON, follows policy, or chooses safer labels. That is why every serious run needs task evals and golden sets alongside loss curves. Compare base vs each checkpoint on the same rubric before and after fine-tuning.

Use checkpoints as candidates

A fine-tuning job can save checkpoints along the way. The last checkpoint is not automatically the best. Earlier checkpoints often generalize better because they learned the pattern before memorizing quirks.

Evaluate multiple checkpoints on the same held-out set. Track exact model ID, adapter ID, dataset version, hyperparameters, and eval score. You want to be able to say "checkpoint 800 won because it improved extraction accuracy by 6 points and did not regress refusal tests," not "the last run looked fine."

A minimal eval gate

A practical first gate can be small and strict:

• Task success must improve over the base model with the best prompt.
• Output format validity must not regress.
• Safety, refusal, or policy cases must not regress.
• Latency and cost must stay within the product budget.
• Human review of sampled failures must show understandable mistakes, not new weird behavior.

If a tuned model cannot beat the base model plus prompt on a fixed eval, do not ship it. Improve the dataset, the prompt, or the base model choice.

When training breaks: OOM and loss spikes

CUDA out of memory is the most common self-serve failure. Fixes, in order of least to most invasive:

• Lower per_device_train_batch_size and increase gradient_accumulation_steps to keep effective batch size.
• Shorten max_seq_length or filter outlier-long rows from the dataset.
• Use QLoRA (4-bit base) instead of 16-bit LoRA.
• Lower LoRA rank or drop MLP target modules.
• Enable gradient checkpointing (trades compute for memory).

Loss spikes or NaN usually mean the update step is too aggressive or the data has bad rows:

• Halve learning rate and resume from the last good checkpoint.
• Check for empty assistant messages, broken JSON, or extreme token lengths.
• Confirm loss masking: if labels are wrong, loss can look unstable or trivially low.
• On QLoRA, verify library versions match; mixed precision bugs show up as NaN mid-run.

Log GPU memory, learning rate, and loss per step. When a run fails at step 400, those logs tell you whether to fix data or hyperparameters.

Quick check