I Quantized and Fine-Tuned a 7B Model on a Free GPU. Turns Out That’s the Easy Part.

4-bit quantization, LoRA fine-tuning, and DSPy prompt compilation on a single T4. The part that helped most wasn’t the training.

I was curious about a practical question: how far can you push a local language model on hardware anyone can access for free? Not a toy demo, but something closer to a working QA system: retrieve relevant text, feed it to the model, get a correct answer back. No API calls, no paid compute.

This is what I found.

The Constraint

Llama-2–7B has roughly 6.7 billion parameters. In 16-bit precision, that’s about 12.5 GiB just for the weights, before you account for optimizer states, activations, or anything else. A free Kaggle or Colab notebook gives you a T4 with 16 GB. That doesn’t leave room for training, retrieval, or inference overhead.

QLoRA solves this. It loads the model in 4-bit precision (NF4 format with double quantization), then attaches small trainable adapter matrices to the frozen base. You only update the adapters during training.

A note on the math. You might expect 7B params × 4 bits = ~3.5 GB for a straight 4-bit model. In practice, the working GPU footprint is higher: bitsandbytes stores quantization scales and metadata per block, embeddings and layer norms stay in fp16, the LoRA adapter weights are full precision, and PyTorch allocates CUDA buffers for inference. The notebook measured 6.32 GiB total with the adapter loaded and the entire model pinned to GPU. That’s roughly half the 16-bit footprint and fits comfortably on a free T4.

With the quantized model and adapter together using ~6.3 GiB, there’s enough headroom on the T4 for a retrieval encoder, tokenizer, and inference buffers.

Fine-Tuning

I fine-tuned on 600 examples from SQuAD, the Stanford question answering benchmark. Each example is a context paragraph, a question, and the shortest text span that answers it. Training ran for 80 gradient steps, about 75 minutes on the T4.

The loss drop is modest. But the model did pick up the extractive QA pattern: given a context paragraph, find and return the relevant span.

The RAG Pipeline

With the fine-tuned model ready, I built a retrieval-augmented pipeline using DSPy. The architecture is straightforward:

The baseline result, with no prompt optimization, was a token-level F1 of 0.419 on 50 held-out questions. 19 out of 50 got perfect scores. But 25 scored zero. The model often had the right answer in its output but either kept generating past it or didn’t format things in a way the system could parse.

The gap: the model knew the answers. The pipeline couldn’t reliably get them out.

Prompt Compilation

DSPy has a concept called prompt compilation. The optimizer I used, BootstrapFewShot, works like this: it runs the model on training examples, identifies cases where the output was both correct and properly formatted, and injects those as few-shot demonstrations into future prompts.

The model weights don’t change. Only the prompt changes.

After compilation, the same model scored 0.813 F1. 37 out of 50 perfect answers.

What Mattered More Than Expected

Shorter outputs beat longer onesThe fine-tuned model loves to keep talking. Capping max_tokens at 64 instead of 256 was the single biggest reliability improvement. The right answer is almost always in the first few tokens.Fewer demonstrations, not moreFirst attempt used 4 bootstrapped + 8 labeled demos. The prompt got so long it pushed the actual question out of the model's context window. Dropping to 2+2 took F1 from near-zero to 0.813.Truncation directionWhen the prompt overflows, cut from the beginning (old demos), not the end (current question). One line: truncation_side="left".A forgiving parserSmall models don't reliably emit structured field markers. A fallback that grabs the first non-empty line as the answer turned crashes into usable predictions.

Takeaway

Getting a 7B model to fit on a free GPU is basically a solved problem. QLoRA handles that cleanly. The harder and more interesting challenge is getting reliable, structured output from a small model inside a real pipeline.

Fine-tuning gives the model knowledge. Prompt compilation gives the pipeline reliability. In this experiment, the second part contributed more than the first, and it required zero additional training. Just a smarter way of asking.

The entire notebook runs end-to-end from a fresh kernel in about 90 minutes.