Fine-Tuning & Evaluation of Small Language Models for Customer Support.

One of the most interesting experiments I recently worked on was:

Fine-Tuning & Evaluation of Small Language Models for Customer Support.

Problem Statement

Most customer-support chatbots today rely on Large Language Models (LLMs), but they can become expensive, slower, and harder to deploy privately for smaller businesses.

The experiment was to explore whether a Small Language Model (SLM) could be fine-tuned effectively for a real-world e-commerce customer-support use case.

Tech Required

• Qwen2.5–3B-Instruct • LoRA (PEFT) Fine-Tuning • HuggingFace Transformers • FP16 Inference • RAGAS Evaluation Framework • BERTScore • Python + Jupyter Notebooks

Experiment

I fine-tuned Qwen2.5–3B-Instruct using LoRA on an e-commerce instruction dataset.

The experiment included multiple stages:

1. Fine-Tuning :

• Trained the SLM on domain-specific customer-support data
• Preserved general model knowledge while injecting business-specific behavior using PEFT/LoRA adapters

2. Environment Replication :

• Created the exact same environment used by the chatbot system for fair comparison

3. Knowledge Base Setup :

• Added product details, shipping policies, refund information, sourcing details, and company information as structured chunks

4. Evaluation :

• Compared generated responses using: RAGAS is an open source framework which is used to evaluate the responses. It automates the process of evaluating RAG systems. It uses metrics to evaluate like: (a) Generation Quality Metrics , (b) Retrieval Quality Metrics , (c) Grounding / Faithfulness Metrics

• BERTScore semantic evaluation: BERTScore is a metric used to evaluate how good an AI-generated answer is by comparing it to a reference answer.It understands meaning, not just exact words.

• baseline vs fine-tuned outputs

Conclusion / What I Learned

By this experiment I learned that Small Language Models are far more capable than many people assume when they are properly fine-tuned for a focused use case.

Some major observations: • significantly lower compute cost • faster inference • easier deployment • better privacy control • surprisingly strong domain adaptation

The fine-tuned model produced responses that were noticeably more aligned with the business context and conversational requirements compared to the baseline model.

Experiment Flow: LLM vs. Fine-Tuned SLM Comparison:

1. Fine-Tune SLM (Qwen2.5–3B-Instruct via LoRA on E-commerce Dataset) ↓
2. Replicate Chatbot Environment & Knowledge Base Setup ↓
3. Generate LLM Baseline Response (Parallel to: Generate Fine-Tuned SLM Response) ↓
4. Evaluate Both Responses (RAGAS & BERTScore Frameworks) ↓
5. Conclusion: Fine-Tuned SLM Outperformed LLM Baseline (Lower Cost, Faster Inference, Stronger Domain Adaptation)

Meet Skynift: AI Built for Business The Skynift Ecosystem:

Skynift is building the future of AI-powered business tools, it stands out by building two deeply integrated products under one ecosystem — customer support and ad performance.

Helpexa makes AI customer support simple and smart for WhatsApp, Instagram, Shopify, and more .

Drac.ai automatically optimizes your ad spend across Google and Meta .