Domain-Specific Small Language Models (SLMs) in Python: Fine-Tuning Phi-3 and Gemma for Industry Use

The AI landscape is shifting fast. For years, the dominant narrative was “bigger is better,” with organizations racing to deploy massive language models that demanded enormous compute budgets and complex infrastructure. That story is changing. Small Language Models, or SLMs, are quietly becoming the workhorses of real-world AI deployments, and for very good reason.

This post explores how you can fine-tune two of the most capable SLMs available today, Microsoft’s Phi-3 and Google’s Gemma, to build domain-specific AI systems in Python. Whether you are a developer at a hospital, a data scientist at a bank, or a tech lead at a logistics company, the ability to train a compact model on your own data is now within reach. We will cover the architecture, the tooling, production-grade code, and the industries already benefiting from this approach.

What Are Small Language Models, and Why Should You Care?

Let us set the record straight. A Small Language Model is not just a shrunken version of GPT-4. SLMs like Phi-3-mini (3.8 billion parameters) and Gemma-2B are purpose-engineered for efficiency. They are trained on high-quality, curated datasets and designed to run on hardware you actually own, including laptops, edge servers, and cloud VMs without GPU clusters.

The critical insight here is this: for most real-world industry tasks, you do not need a model that can write poetry in seventeen languages. You need a model that understands your specific domain deeply, responds reliably, and fits inside a production environment with predictable latency. That is exactly what a fine-tuned SLM delivers.

Fine-tuning is the process of taking a pre-trained model and continuing to train it on a smaller, domain-specific dataset. Instead of training from scratch (which would cost millions of dollars), you adapt the model’s existing knowledge to your use case. The result is a model that speaks your industry’s language.

The Core Architecture: How Fine-Tuning Works

Modern SLM fine-tuning relies on a technique called Parameter-Efficient Fine-Tuning, or PEFT. The most popular method within PEFT is LoRA (Low-Rank Adaptation). Here is the idea in plain terms.

A language model has billions of weight matrices. Full fine-tuning updates every single one of these weights, which is expensive and slow. LoRA instead injects small, trainable “adapter” matrices alongside the original frozen weights. During training, only these adapters are updated. During inference, their output is added back to the frozen model’s output. The result is near-full-fine-tune quality at a fraction of the cost.

QLoRA goes one step further. It quantizes the frozen base model to 4-bit precision (reducing its memory footprint dramatically), then applies LoRA on top. This makes it possible to fine-tune a 7-billion-parameter model on a single consumer GPU with 24GB of VRAM.

Key Tools and Frameworks

The Python ecosystem around SLM fine-tuning is mature and well-integrated. Here are the tools you will actually use:

Hugging Face Transformers is the entry point for loading Phi-3 and Gemma. It provides pre-built model classes, tokenizers, and training utilities that abstract away most of the low-level complexity.

PEFT (Parameter-Efficient Fine-Tuning) is the Hugging Face library that implements LoRA, QLoRA, prefix tuning, and other adapter methods. You configure a LoRA config, wrap your model, and start training.

BitsAndBytes handles the 4-bit quantization required for QLoRA. It runs seamlessly on NVIDIA GPUs and allows you to load a 7B model in roughly 4GB of VRAM instead of 14GB.

TRL (Transformer Reinforcement Learning) provides the SFTTrainer class, which is a wrapper around the standard Hugging Face Trainer optimized for supervised fine-tuning on instruction-formatted data. It handles dataset formatting, gradient accumulation, and packing automatically.

Weights and Biases (W&B) is the de facto tool for experiment tracking. Every training run logs your loss curves, learning rate schedules, and evaluation metrics to a dashboard you can share with your team.

LangChain becomes relevant post-fine-tuning. Once your model is specialized, you can embed it inside a LangChain pipeline to add retrieval-augmented generation (RAG), tool use, or multi-step reasoning on top of the fine-tuned base.

Detailed Code Sample

The following code demonstrates fine-tuning Phi-3-mini on a domain-specific instruction dataset using QLoRA and TRL’s SFTTrainer. It is structured to run on a single A100 or equivalent GPU.

# =====================================================================
# Fine-Tuning Phi-3-mini with QLoRA for Domain-Specific Use
# Requirements: transformers, peft, trl, bitsandbytes, datasets, torch
# =====================================================================

import torch
from datasets import load_dataset
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    BitsAndBytesConfig,
    TrainingArguments,
)
from peft import LoraConfig, get_peft_model, TaskType
from trl import SFTTrainer

# ------------------------------------------------------------------
# STEP 1: Configure 4-bit quantization (QLoRA)
# This loads the base model in 4-bit precision, cutting memory ~75%
# ------------------------------------------------------------------
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,                        # Enable 4-bit loading
    bnb_4bit_quant_type="nf4",                # NormalFloat4: best quality for QLoRA
    bnb_4bit_compute_dtype=torch.bfloat16,    # Use bfloat16 for stable training
    bnb_4bit_use_double_quant=True,           # Nested quantization saves more memory
)

# ------------------------------------------------------------------
# STEP 2: Load the base model and tokenizer
# Swap "microsoft/Phi-3-mini-4k-instruct" with "google/gemma-2b-it"
# for Gemma-based fine-tuning. The rest of the code stays identical.
# ------------------------------------------------------------------
MODEL_ID = "microsoft/Phi-3-mini-4k-instruct"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True)
tokenizer.pad_token = tokenizer.eos_token   # Required for batched training
tokenizer.padding_side = "right"            # Prevent gradient issues with causal LM

model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    quantization_config=bnb_config,
    device_map="auto",                        # Automatically maps layers across GPUs
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
)
model.config.use_cache = False              # Disable KV cache during training
model.config.pretraining_tp = 1            # Required for Phi-3 stability

# ------------------------------------------------------------------
# STEP 3: Configure LoRA adapters
# We target the attention projection layers: these are the most
# impactful layers to adapt for new domain knowledge.
# ------------------------------------------------------------------
lora_config = LoraConfig(
    r=16,                                    # Rank: higher = more capacity, more cost
    lora_alpha=32,                           # Scaling factor (typically 2x rank)
    target_modules=[                         # Layers to apply adapters to
        "q_proj", "k_proj", "v_proj",
        "o_proj", "gate_proj",
        "up_proj", "down_proj"
    ],
    lora_dropout=0.05,                       # Light dropout to prevent overfitting
    bias="none",                             # Don't adapt bias terms
    task_type=TaskType.CAUSAL_LM,           # We are doing causal language modeling
)

model = get_peft_model(model, lora_config)

# Print trainable parameter count to verify LoRA is working correctly
trainable, total = model.get_nb_trainable_parameters()
print(f"Trainable parameters: {trainable:,} ({100 * trainable / total:.2f}% of total)")
# Expected output: ~0.5-2% of parameters are trainable with LoRA

# ------------------------------------------------------------------
# STEP 4: Prepare the domain-specific dataset
# For this example we use a medical Q&A dataset.
# Replace with your own dataset in the same format.
# ------------------------------------------------------------------

# Dataset format: each record must have an "instruction" and "response" field.
# For production, load from your own JSONL file or internal database.
dataset = load_dataset("medalpaca/medical_meadow_medical_flashcards", split="train")

def format_instruction(sample):
    """
    Format each sample into Phi-3's chat template.
    Phi-3 uses <|user|> and <|assistant|> tokens natively.
    For Gemma, replace with: f"<start_of_turn>user\n{...}<end_of_turn>\n<start_of_turn>model\n{...}<end_of_turn>"
    """
    return {
        "text": f"<|user|>\n{sample['input']}<|end|>\n<|assistant|>\n{sample['output']}<|end|>"
    }

dataset = dataset.map(format_instruction, remove_columns=dataset.column_names)

# Keep a small validation split to monitor generalization
dataset = dataset.train_test_split(test_size=0.05, seed=42)
train_data = dataset["train"]
eval_data  = dataset["test"]

print(f"Training samples: {len(train_data):,}")
print(f"Validation samples: {len(eval_data):,}")

# ------------------------------------------------------------------
# STEP 5: Define training arguments
# Tuned for a 3.8B model on a single A100-40GB GPU.
# Adjust batch size and gradient accumulation for your hardware.
# ------------------------------------------------------------------
training_args = TrainingArguments(
    output_dir="./phi3-medical-qlora",       # Where checkpoints are saved
    num_train_epochs=3,                      # 3 epochs is a solid baseline
    per_device_train_batch_size=4,           # Keep low to fit in 4-bit VRAM
    per_device_eval_batch_size=4,
    gradient_accumulation_steps=4,           # Effective batch size = 4 * 4 = 16
    gradient_checkpointing=True,             # Trade compute for memory savings
    optim="paged_adamw_32bit",              # Memory-efficient optimizer for QLoRA
    learning_rate=2e-4,                      # Higher LR works well with LoRA
    lr_scheduler_type="cosine",              # Cosine decay is smoother than linear
    warmup_ratio=0.03,                       # 3% warmup prevents initial instability
    weight_decay=0.001,
    fp16=False,                              # Use bf16 instead of fp16
    bf16=True,                               # bfloat16: better range than float16
    max_grad_norm=0.3,                       # Clip gradients to prevent spikes
    logging_steps=25,
    evaluation_strategy="steps",
    eval_steps=100,
    save_strategy="steps",
    save_steps=200,
    save_total_limit=3,                      # Keep only last 3 checkpoints
    load_best_model_at_end=True,
    report_to="wandb",                       # Log to Weights and Biases
    run_name="phi3-medical-qlora-v1",
)

# ------------------------------------------------------------------
# STEP 6: Initialize the SFTTrainer and start training
# SFTTrainer handles dataset packing and formatting automatically.
# ------------------------------------------------------------------
trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=train_data,
    eval_dataset=eval_data,
    tokenizer=tokenizer,
    dataset_text_field="text",               # Column containing formatted text
    max_seq_length=2048,                     # Max context window for Phi-3-mini
    packing=True,                            # Pack multiple short samples for efficiency
)

print("Starting fine-tuning...")
trainer.train()
print("Fine-tuning complete.")

# ------------------------------------------------------------------
# STEP 7: Save the LoRA adapters and optionally merge with base model
# You can serve adapters alone (faster loading) or merge for simplicity.
# ------------------------------------------------------------------
ADAPTER_DIR = "./phi3-medical-adapter"
trainer.model.save_pretrained(ADAPTER_DIR)
tokenizer.save_pretrained(ADAPTER_DIR)
print(f"LoRA adapters saved to {ADAPTER_DIR}")

# Optional: merge adapters back into the base model for deployment
# This creates a standalone model without the PEFT dependency at runtime.
from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)
merged_model = PeftModel.from_pretrained(base_model, ADAPTER_DIR)
merged_model = merged_model.merge_and_unload()    # Bakes adapters into base weights

MERGED_DIR = "./phi3-medical-merged"
merged_model.save_pretrained(MERGED_DIR)
tokenizer.save_pretrained(MERGED_DIR)
print(f"Merged model saved to {MERGED_DIR}")

# ------------------------------------------------------------------
# STEP 8: Quick inference test on the fine-tuned model
# ------------------------------------------------------------------
def generate_response(model, tokenizer, question, max_new_tokens=256):
    prompt = f"<|user|>\n{question}<|end|>\n<|assistant|>\n"
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            temperature=0.3,          # Lower temperature for factual domain tasks
            top_p=0.9,
            do_sample=True,
            pad_token_id=tokenizer.eos_token_id,
        )

    response = tokenizer.decode(outputs[0], skip_special_tokens=True)
    return response.split("<|assistant|>")[-1].strip()

# Test with a domain-specific question
test_question = "What are the first-line treatments for Type 2 diabetes?"
answer = generate_response(merged_model, tokenizer, test_question)
print(f"Q: {test_question}")
print(f"A: {answer}")

Pros of Domain-Specific SLM Fine-Tuning

Lower operational cost. A fine-tuned 3.8B model can replace a GPT-4 API call for narrow tasks, reducing inference costs by orders of magnitude, especially at scale.

Data stays on-premise. Unlike API-based LLMs where your prompts leave your network, a locally deployed SLM processes everything internally. This is non-negotiable for regulated industries.

Significantly higher domain accuracy. A generic LLM hallucinates on niche terminology. A fine-tuned SLM trained on your data speaks your language and dramatically reduces error rates on domain-specific tasks.

Faster inference at lower latency. A 4-bit quantized 3.8B model can run at 40 to 80 tokens per second on a single A10G GPU. That is production-ready latency for most applications.

Full model ownership. You own the weights. You decide when to update, retrain, or retire the model. There is no dependency on a third-party provider’s API uptime or pricing changes.

Edge and offline deployment. Phi-3-mini can run on a laptop CPU using llama.cpp. Gemma-2B can be deployed on a Raspberry Pi 5. This opens genuinely new categories of offline-capable AI applications.

Regulatory compliance is achievable. Fine-tuned SLMs can be audited, version-controlled, and evaluated against compliance benchmarks in a way that black-box API models simply cannot match.

Parameter-efficient training is accessible. QLoRA makes fine-tuning a 7B model accessible on a single consumer GPU with 24GB VRAM, meaning the barrier to entry is genuinely low.

Industries Using Domain-Specific SLMs

Healthcare

Hospitals and clinical software companies are fine-tuning SLMs on clinical notes, ICD-10 coding guidelines, and medical literature. A fine-tuned Phi-3 can assist radiologists by summarizing CT reports, help nurses draft discharge summaries from structured EHR data, or power a clinical decision support tool that flags drug interactions. The HIPAA compliance requirement makes local deployment a strict necessity, which is exactly where fine-tuned SLMs shine.

Finance

Banks and asset managers are using fine-tuned Gemma models for earnings call analysis, regulatory filing summarization (10-K, 10-Q), and internal policy Q&A bots. A model fine-tuned on SEC filings and internal compliance documents can answer questions like “does this transaction require SAR filing?” with much higher accuracy than a generic LLM. Risk and compliance teams also appreciate that no client data needs to leave their secure environment.

Retail and E-commerce

Product catalog intelligence is a major use case. Retailers fine-tune SLMs on their product descriptions, return policies, and customer service transcripts to power support chatbots that actually understand their catalog. A model trained on thousands of past customer interactions can handle nuanced queries like “will this jacket fit if I usually wear a medium in Nike?” with genuine contextual understanding.

Automotive

OEMs and Tier-1 suppliers are fine-tuning SLMs on technical service bulletins, repair manuals, and diagnostic trouble codes (DTCs). A technician chatbot that has been trained on the full maintenance history and service documentation for a specific vehicle platform can guide mechanics through complex diagnostics in natural language, reducing repair times and warranty claims.

Legal

Law firms and legal tech companies are fine-tuning SLMs on jurisdiction-specific case law, contract templates, and regulatory texts. A model fine-tuned on UK employment law can help associates quickly find relevant precedents, draft standard clauses, or flag risky language in new contracts. The confidentiality of client matters makes on-premise deployment the only viable option for most firms.

How PySquad Can Assist

PySquad has deep, hands-on expertise in the full SLM fine-tuning lifecycle, from data curation all the way to production deployment. Here is why teams trust PySquad with this kind of critical AI work:

• **PySquad has delivered fine-tuning projects across multiple industries**, including healthcare, fintech, and legal tech, building domain-specific models that outperform generic LLM APIs on narrow, high-stakes tasks.
• **PySquad’s engineering team has direct experience** with QLoRA pipelines, PEFT configurations, and model quantization, meaning there is no guesswork in the implementation. Every architectural decision is backed by real experimentation.
• **PySquad provides end-to-end ownership** of your fine-tuning project, from dataset preparation and cleaning, through training and evaluation, to deployment on your chosen infrastructure, whether that is cloud, on-premise, or edge.
• Data privacy is a first-class concern at PySquad. Every project is structured so that sensitive domain data never leaves your environment. PySquad builds training and inference pipelines that run entirely within your security perimeter.
• **PySquad runs rigorous evaluation benchmarks** on every fine-tuned model before delivery, including domain-specific accuracy tests, hallucination rate measurement, and latency profiling under realistic load.
• **PySquad has worked with Hugging Face Transformers, TRL, PEFT, BitsAndBytes, and the full modern fine-tuning stack**, so integration with your existing Python infrastructure is seamless rather than disruptive.
• **PySquad’s team stays current with the fastest-moving area of AI.** When new model architectures like Phi-3.5 or Gemma-3 drop, PySquad evaluates them immediately and can advise on whether upgrading your fine-tuned model is worth the migration cost.
• **PySquad delivers production-grade MLOps alongside the model itself**, including model versioning, A/B testing infrastructure, monitoring for distribution shift, and automated retraining pipelines.
• **PySquad is transparent about trade-offs.** If your use case is better served by RAG over a general model rather than fine-tuning, PySquad will tell you that clearly. The goal is the best outcome for your product, not the most expensive engagement.
• Working with PySquad means you build internal capability, not just a deliverable. PySquad’s engagements include knowledge transfer, documentation, and training sessions so your team understands and can maintain what has been built.

References

1. Hugging Face PEFT Documentation — The official guide to Parameter-Efficient Fine-Tuning methods including LoRA and QLoRA: https://huggingface.co/docs/peft
2. QLoRA: Efficient Finetuning of Quantized LLMs (Dettmers et al., 2023) — The original paper introducing QLoRA, available on arXiv: https://arxiv.org/abs/2305.14314
3. Microsoft Phi-3 Technical Report — Microsoft’s official report on the Phi-3 model family, covering architecture, training data, and benchmarks: https://arxiv.org/abs/2404.14219
4. Google Gemma Model Card and Documentation — Official Gemma documentation on Hugging Face Hub: https://huggingface.co/google/gemma-2b-it
5. TRL (Transformer Reinforcement Learning) GitHub Repository — The SFTTrainer and training utilities used in this post: https://github.com/huggingface/trl

Conclusion

If there is one thing to take away from this post, it is that domain-specific AI is no longer the exclusive territory of organizations with massive AI budgets and research teams. Fine-tuning Phi-3 or Gemma on your own data, using QLoRA and the Hugging Face ecosystem in Python, is a genuinely practical engineering task that produces real, measurable business value.

We covered the architectural foundation of LoRA and QLoRA, walked through a production-ready fine-tuning pipeline with inline explanations, explored how industries from healthcare to legal are putting these models to work, and outlined the concrete advantages that come from owning your own fine-tuned model rather than depending on an API.

The next steps are straightforward. Clone the code sample, swap in your own dataset, and run a first training experiment. Start with Phi-3-mini if you want to move fast on CPU-accessible hardware, or Gemma-2B-it if you want strong instruction-following out of the box. Track your experiments with Weights and Biases and evaluate your model honestly on domain-specific held-out data before you declare success.