Inference, RAG, Fine-Tuning, or Training: Which LLM Workload Are You Running?

A team decides to “build an LLM,” then immediately starts comparing GPUs.

But what does “build an LLM” actually mean?

You may only need to run an existing model. You might need to connect it to company documents, adjust how it responds, or train an entirely new model.

Each workload requires different data, tools, time, and GPU capacity.

Which LLM Workload Matches Your Goal?

Most developers and small teams need inference, RAG, or fine-tuning.

Very few need to train an LLM from scratch.

1. Inference: You Want to Use an Existing Model

Inference means giving a trained model an input and receiving an output.

Examples include:

• Generating an answer
• Summarizing a document
• Classifying a support ticket
• Writing code
• Extracting information
• Translating text

The model’s weights are not updated. You are simply using what the model has already learned.

Choose inference when:

• An existing model already performs the task
• Prompting gives acceptable results
• You are building an early prototype
• You do not need private or frequently updated knowledge
• You want to test model speed, quality, or cost

Example

You want to use a 7B instruction model to summarize customer calls.

You provide the transcript, prompt the model, and receive a summary. That is inference.

Typical GPU requirement

Inference usually requires less GPU memory than fine-tuning or training.

The actual requirement depends on:

• Model size
• Precision or quantization
• Context length
• Number of simultaneous users
• Required response speed

Start with inference before adding more complicated workflows.

2. RAG: The Model Needs Access to Your Information

Retrieval-augmented generation, or RAG, retrieves relevant information from an external source and adds it to the model’s prompt before generating an answer.

The model itself is not retrained.

A basic RAG flow looks like this:

User question

↓

Search relevant documents

↓

Add the retrieved content to the prompt

↓

LLM generates an answer

Choose RAG when:

• Answers must use company documents
• Information changes regularly
• Users need source-grounded responses
• The model must search policies, manuals, tickets, or product data
• Updating model weights whenever information changes would be impractical

Example

You are building a support assistant that must answer from:

• Product documentation
• Release notes
• Troubleshooting guides
• Previous support tickets

When the documents change, you update the knowledge base instead of fine-tuning the model again.

Important limitation

RAG gives the model information. It does not permanently teach the model that information.

Poor retrieval can still produce poor answers. You must test document chunking, embeddings, retrieval quality, and prompt construction.

Image Credit: Hugging Face Open-Source AI Cookbook.

3. Fine-Tuning: You Want to Change How the Model Behaves

Fine-tuning continues training an existing model on a smaller, task-specific dataset.

Unlike RAG, fine-tuning updates model parameters.

Choose fine-tuning when you need:

• A consistent tone or writing style
• Reliable structured output
• Better performance on a narrow task
• Domain-specific terminology
• A specific response pattern
• Behaviour that prompting cannot produce consistently

Example

You need the model to convert support conversations into this exact format:

{
"issue": "",
"priority": "",
"recommended_action": ""
}

Prompting works most of the time, but the model often changes field names or adds extra text.

A carefully prepared fine-tuning dataset may improve consistency.

Full fine-tuning vs LoRA

Full fine-tuning updates most or all model parameters and can require substantial GPU memory.

Parameter-efficient methods such as LoRA train a much smaller set of additional parameters. This lowers compute and storage requirements and is often more practical for small teams.

What fine-tuning is not good for

Fine-tuning is usually not the best way to keep frequently changing facts updated.

Do not fine-tune a model every time:

• A price changes
• A policy is updated
• A new product is launched
• Documentation is edited

Use RAG for changing knowledge. Use fine-tuning for persistent behaviour.

4. Training from Scratch: You Want to Create the Base Model

Training from scratch means starting with randomly initialized model weights and teaching the model using a large dataset.

This is how base models are created before instruction tuning or task-specific fine-tuning.

Choose training from scratch only when:

• Existing models cannot meet your requirements
• You have a large, suitable dataset
• You have distributed-training expertise
• You can afford extensive experimentation
• You need full control over the model architecture and training data

For most developers and startups, this is unnecessary.

Training a useful base LLM may require:

• Large datasets
• Multiple high-memory GPUs
• Distributed training
• Data cleaning and deduplication
• Checkpoint management
• Repeated evaluation
• Considerable engineering time

Fine-tuning begins with an already pretrained model, which requires much less compute, data, and time than starting from random weights.

Can You Use More Than One Workload?

Yes. These are not always separate choices.

A production application may use all three:

1. Fine-tune the model to follow a specific output format.
2. Use RAG to retrieve current business information.
3. Run inference whenever a user sends a request.

Inference still happens at the end, even when you use RAG or a fine-tuned model.

Example

A legal-document assistant could:

• Use fine-tuning to produce a consistent analysis format
• Use RAG to retrieve relevant contracts and policies
• Use inference to generate the final response

A Simple Way to Choose

Ask what is currently wrong with the model.

“The model already gives useful answers.”

Use inference.

“The model does not know our documents or current information.”

Use RAG.

“The model knows enough, but its behaviour is inconsistent.”

Consider fine-tuning.

“No existing base model is suitable.”

Only then investigate training from scratch.

Common Mistakes

Fine-tuning when the problem is missing knowledge

Giving the model current documents through RAG is usually easier than repeatedly retraining it.

Building RAG when better prompting is enough

Test the base model and prompt before adding a vector database and retrieval pipeline.

Training before creating a baseline

You cannot measure improvement unless you know how the original model performs.

Confusing RAG with inference

RAG retrieves information, but the model still performs inference to generate the final answer.

Choosing hardware before defining the workload

A GPU suitable for quantized inference may not have enough memory for full fine-tuning.

Image Credit: GIF by Pudgy Penguins, via GIPHY.

Start With the Least Complex Option

Start with the least complicated workload.

• Use inference when an existing model already solves the task.
• Use RAG when the model needs access to private or changing information.
• Use fine-tuning when you need consistent behaviour or task performance.
• Consider training from scratch only when existing models are fundamentally unsuitable.

For most small teams, the practical order is:

Prompting → inference test → RAG if knowledge is missing → fine-tuning if behaviour remains inconsistent

Do not train more than the problem requires.

Helpful Resources

1. Hugging Face text-generation guide
2. Hugging Face simple RAG tutorial
3. Hugging Face fine-tuning guide
4. Hugging Face PEFT documentation
5. How to fine-tune an open-source LLM without overspending