Do you really want to fine-tune your LLM?

At a recent workshop in London, I had the chance to hear OpenAI’s perspective on strategies for refining large language model (LLM) behavior. Their breakdown was sharp and illuminating, enough to spark a moment of clarity I felt worth sharing. While some of the code snippets echo concepts introduced during the workshop, most of the examples in this article reflects my own exploration and of course, some FUN coding.

Optimization of LLM solutions

Let us begin by discussing the optimization strategies illustrated in the familiar image above. Context optimization shapes the information available to the model at inference time. It is achieved through methods such as prompt engineering or retrieval-augmented generation (RAG). While this approach leaves the model itself unchanged, its outputs improve because it is given more relevant input. LLM optimization, in contrast, modifies the parameters or structure of the model directly. Fine-tuning and distillation fall into this category, producing models that are better aligned with specific domains or tasks.

The two approaches address different trade‑offs. Context optimization provides flexibility and efficiency, while LLM optimization delivers deeper specialization at higher computational cost. The strongest systems emerge when both dimensions are combined, layering prompt design, RAG, and fine‑tuning into a unified optimization flow. For instance, you might begin by shaping prompts, so the model consistently follows instructions and produces structured outputs. Next, you add RAG to supply the model with domain‑specific documents at runtime, ensuring its answers are grounded in current and relevant knowledge. Finally, you fine‑tune the model on curated examples from your domain, so it develops a built‑in familiarity with the style, terminology, and edge cases of your use case. The result is a system that responds reliably to instructions, draws on accurate external context, and carries domain expertise natively.

With that foundation in place, let’s explore what fine-tuning really entails and when it’s worth the effort.

Fine-tuning generative AI models

I’d like to dive a bit deeper into fine-tuning OpenAI models, since exploring it surfaced some tools and techniques I had overlooked when RAG first became the go-to option. Before resorting to the heavy lift of fine-tuning as an optimization strategy, make sure you’ve thoroughly explored and tested other approaches:

• Your base model may already perform well with prompt engineering and RAG. Fine-tuning is not a shortcut for poor retrieval or unclear prompts.
• You have a clearly defined gap that RAG or prompt design cannot bridge (e.g., domain-specific tone, structured outputs, or consistent formatting).
• You’ve gathered a high-quality, representative dataset. Fine-tuning amplifies both strengths and flaws, so noisy or inconsistent data will backfire. Ensure that you identify the right examples for a specific use case.
• The cost–benefit tradeoff makes sense. Fine-tuning adds training and maintenance overhead, so it should deliver measurable improvements in accuracy, efficiency, or user experience.
• You’ve validated with smaller experiments first. Make sure you build robust evaluation pipelines as sometimes few-shot prompting can achieve most of the benefit with less complexity.

If you’re still leaning toward fine-tuning your LLM, here are some practical approaches that could move you closer to your desired outcomes.

Fine-tuning methods

Supervised fine-tuning (SFT)

This method helps LLMs adapt to perform specific tasks by training it on a curated dataset of labeled input-output pairs. Unlike general pretraining, which teaches the model broad language patterns, SFT teaches it how to follow instructions, mimic desired behavior, or specialize in a domain. When you want the model to apply specific formatting or tone, you must start by creating the training data including structured outputs, domain-specific terminology, or edge cases.

Imagine you work at a company that provides answers questions about compliance and security. You want the LLM to respond like your best support agent, politely, accurately, and to be familiar with the subject.

Collect examples: You gather a dataset of common compliance and security inquiries and ideal responses written by your top agents. Each entry looks like:

• Input: “What are the compliance requirements for handling personal data?”
• Output: “To handle personal data responsibly, organizations must comply with frameworks like the GDPR in the EU and the CCPA in the US, which emphasize transparency, data minimization, and user rights. For a deeper dive into these principles and how they apply across jurisdictions, explore GDPR guidance from the European Commission and CCPA details from the California Attorney General’s office. References: Data protection — European Commission, California Consumer Privacy Act (CCPA) | State of California — Department of Justice — Office of the Attorney General”

{"messages": [{"role": "system", "content": "You are a helpful assistant that answers questions about compliance and security."}, {"role": "user", "content": "What are the compliance requirements for handling personal data?"}, {"role": "assistant", "content": "To handle personal data responsibly, organizations must comply with frameworks like the GDPR in the EU and the CCPA in the US, which emphasize transparency, data minimization, and user rights. For a deeper dive into these principles and how they apply across jurisdictions, explore GDPR guidance from the European Commission and CCPA details from the California Attorney General's office.\nReferences: Data protection - European Commission, California Consumer Privacy Act (CCPA) | State of California - Department of Justice - Office of the Attorney General"}]}
{"messages": [{"role": "system", "content": "You are a helpful assistant that answers questions about compliance and security."}, {"role": "user", "content": "What is the principle of data minimization?"}, {"role": "assistant", "content": "Data minimization means collecting only the personal data that is necessary for the specific purpose you're processing it for. This principle is a core requirement under GDPR Article 5(1)(c) and helps reduce privacy risks by limiting the amount of data at risk in case of a breach.\nReferences: GDPR Article 5 - Principles relating to processing of personal data"}]}
{"messages": [{"role": "system", "content": "You are a helpful assistant that answers questions about compliance and security."}, {"role": "user", "content": "How long do I need to notify authorities about a data breach?"}, {"role": "assistant", "content": "Under GDPR, you must notify the relevant supervisory authority within 72 hours of becoming aware of a personal data breach, unless the breach is unlikely to result in a risk to individuals' rights and freedoms. Many other regulations have similar timelines, so check your specific jurisdiction's requirements.\nReferences: GDPR Article 33 - Notification of a personal data breach to the supervisory authority"}]}
{"messages": [{"role": "system", "content": "You are a helpful assistant that answers questions about compliance and security."}, {"role": "user", "content": "What is a Data Protection Impact Assessment (DPIA)?"}, {"role": "assistant", "content": "A DPIA is a process to identify and minimize data protection risks in projects that are likely to result in high risks to individuals' privacy. Under GDPR Article 35, DPIAs are mandatory for certain types of processing, such as large-scale profiling or processing of sensitive data categories.\nReferences: GDPR Article 35 - Data protection impact assessment"}]}
{"messages": [{"role": "system", "content": "You are a helpful assistant that answers questions about compliance and security."}, {"role": "user", "content": "What are the key differences between GDPR and CCPA?"}, {"role": "assistant", "content": "GDPR applies to EU residents and requires explicit consent for data processing, while CCPA applies to California residents and uses an opt-out model. GDPR has broader territorial scope and stricter requirements, whereas CCPA focuses on consumer rights like disclosure, deletion, and opt-out of data sales.\nReferences: GDPR official text, CCPA official text from California Legislative Information"}]}

Train the model: You fine-tune the LLM on hundreds or thousands of these input–output pairs. The model learns to mimic the tone, structure, and troubleshooting logic.

Test and Evaluate: Now, when a user asks a similar question, the fine-tuned model responds with tailored advice.

Via SFT, the model learns to internalize domain knowledge and response style.

• Refer to Microsoft’s documentation for non-OpenAI models.

Model distillation

One way to provide data for fine-tuning a LLM is through model distillation. Via this technique a smaller model is trained using the output from more capable models to achieve high performance on specific use cases using a lower cost model. For example, a gpt-4.1-nano model could be trained on a gpt-4.1 model’s output.

• Read more in the documentation of OpenAI.

Vision fine-tuning

Vision fine-tuning is the process of adapting a multimodal language model to better understand and respond to visual inputs by training it on image–text pairs. Just like you would fine-tune an image classification model using labeled images, here you provide:

• Input: an image (e.g., a product photo, chart, or UI screenshot)
• Output: the desired textual response (e.g., a caption, diagnosis, or explanation)

The model learns to associate visual patterns with specific language outputs, improving its performance on specific tasks.

• OpenAI example
• Microsoft example

Direct Preference Optimization (DPO)

This method teaches a language model to prefer certain outputs over others, based on human or system-defined preferences. Each training example includes:

• Input: a user prompt
• Output A: preferred response
• Output B: non-preferred response

The model learns to assign higher likelihood to the preferred output, aligning its behavior with human values, tone, or task-specific goal. This technique is a good approach when you can collect preference data from in-product feedback loops to continuously improve the performance of the model.

• OpenAI example
• Microsoft example

Reinforcement fine-tuning (RFT)

This method improves the model’s performance on tasks with clear definitions and minimal variations, such as math problems, code generation, or structured question answering.

• OpenAI example
• Microsoft example

Start a fine-tuning job

When fine-tuning with Azure AI Foundry, begin by uploading your training file to Azure OpenAI Files. Once the file is in place, you can initiate a fine-tuning job directly through the Foundry interface.

# Upload the training file
print("Uploading training file...")
with open("data.jsonl", "rb") as f:
    training_file = client.files.create(
        file=f,
        purpose="fine-tune"
    )

print(f"Training file uploaded. File ID: {training_file.id}")

# Start fine-tuning job
job = client.fine_tuning.jobs.create(
    model=model_name,
    training_file="file-9c523bc034ae4f119721e1c2a6127673",
    hyperparameters={
        "n_epochs": 3
    }
)
print("Fine-tuning job created. Job ID:", job.id)

Progress and results can be monitored both within Azure AI Foundry and programmatically by executing the below code:

# Retrieve specific job
job = client.fine_tuning.jobs.retrieve(job.id)
print("Status:", job.status)
print("Fine-tuned model:", job.fine_tuned_model)

Fine-tuning logs on Azure AI Foundry

After the job completes (typically after a few hours), deploy the fine-tuned model and proceed with testing and evaluation following the same methodology used for the base model to ensure consistent comparison.

1. Deploy model

import json
import requests

token = ""
resource_group = ""
resource_name = ""
subscription = ""
finetuned_model = "gpt-4.1-nano-2025-04-14.ft-0eb9b13bbfac456c8c0fbe1847ad6d06"

deploy_params = {'api-version': "2025-07-01-preview"} 
deploy_headers = {'Authorization': 'Bearer {}'.format(token), 'Content-Type': 'application/json'}

deploy_data = {
    "sku": {"name": "developertier", "capacity": 50},
    "properties": {
        "model": {
            "format": "OpenAI",
            "name": finetuned_model,
        }
    }
}
deploy_data = json.dumps(deploy_data)

request_url = f'https://management.azure.com/subscriptions/{subscription}/resourceGroups/{resource_group}/providers/Microsoft.CognitiveServices/accounts/{resource_name}/deployments/{finetuned_model}'

print('Creating a new deployment...')

r = requests.put(request_url, params=deploy_params, headers=deploy_headers, data=deploy_data)

print(r)
print(r.reason)
print(r.json())

Read more about model deployment in the Azure OpenAI Documentation.

Deployment overview

2. Evaluate model

Fine-tuning and evaluation are inseparable — without rigorous evaluation, optimization efforts risk becoming guesswork. Let’s quantify gains in relevance and factuality. Read more about evaluation in my other article: Evaluate before and after fine-tuning!

Results of the base model:

Results of the fine-tuned model:

While there are minimal differences between the two evaluation scores, the output of the fine-tuned LLM is as expected by the application design, while the base model responds in a lengthy, less concise way.

Output of the base model

Output of the fine-tuned model

You can find the complete codebase in this repository on GitHub.

Conclusion

Fine-tuning can unlock powerful capabilities in large language models but it’s not a silver bullet. The tradeoffs are real: increased specialization comes with added complexity, cost, and the risk of overfitting, especially when working with small or noisy datasets. A fine-tuned model will amplify the strengths of your data but also its flaws.

The most effective systems often emerge not from brute-force optimization, but from thoughtful layering of context, retrieval, and targeted refinement.

Other References

Model optimization — OpenAI API

Fine-tune models with Azure AI Foundry — Azure AI Foundry | Microsoft Learn