How to Use LLMs in Real-World Engineering, and Why Small Language Models Are the Key to Success

Over the past year, Large Language Models (LLMs) have moved from being an experimental curiosity to a genuine productivity enhancer for engineering teams. Yet, as engineers, we often discover that real-world implementations aren’t as glamorous as a ChatGPT demo, the friction comes in aligning these models with our messy, evolving codebases and processes.

Recently, I stumbled upon a problem that almost every developer has felt: 👉 “Why am I reviewing the same damn thing on multiple PRs?”

Our internal PR review bot wasn’t helping. It flagged outdated issues, missed newer guidelines, and, worse, had no memory of evolving team conventions. It became clear: LLMs and AI agents are only as good as the knowledge they can access.

Not able to read whole article ? Read here [medium link]

The Problem: Static Bots in a Dynamic Codebase

At my workplace, we rely on bots to automate pull request checks. In theory, they should free us from repetitive nitpicks. In practice, they became stale quickly.

• Code guidelines changed, but the bot wasn’t updated.
• Team discussions uncovered new best practices, but the bot had no awareness.
• Our monolithic codebase (50+ GB) was too large for any LLM to crawl meaningfully.

So while bots flagged lint errors or type mismatches, I was still commenting on the same coding patterns over and over again.

The Breakthrough: An AI Agent With Context

Instead of forcing a huge LLM to digest our entire repo, I decided to try something different: build a focused AI agent with relevant context only.

Here’s what I did:

1. Connected the Agent to Our Internal Wiki

• This gave the agent access to living documentation — guidelines, architecture decisions, PR examples, and discussions.
• Unlike static config files, our wiki evolves continuously.

2. Fed Git Diffs Instead of Entire Codebases

• Every PR generates a git diff.
• Instead of sending the AI 50 GB of code, I sent just the relevant delta.
• The agent could now analyze changes in the context of the wiki knowledge.

3. Created a Feedback Loop

• When I commented on a PR — e.g., “We should only use React.memo when absolutely necessary” — that knowledge was logged into the wiki.
• Now, the agent could reuse this insight when reviewing another PR with the same anti-pattern.

The result? The AI started surfacing relevant, context-aware suggestions, directly tied to our team’s evolving practices.

Example: React.memo in PR Reviews

Here’s a simple story that shows how impactful this setup became:

• A teammate submitted a PR where React.memo was overused.
• I added a comment explaining why indiscriminate memoization can actually harm performance.
• That comment was saved to the wiki.
• A week later, another teammate made a similar PR. This time, the AI agent caught it and suggested the same reasoning upfront.

Suddenly, the AI wasn’t just a static rule checker. It had become an active participant in our knowledge loop, learning from PRs and reinforcing team conventions.

Why Small Language Models (SLMs) Matter

This experience highlighted an important truth: bigger isn’t always better.

• LLMs (like GPT-4, Claude, etc.) are powerful but expensive to run on large enterprise codebases. They also raise security and compliance concerns when dealing with sensitive repos.
• SLMs (Small Language Models), when scoped properly, can outperform LLMs in specific contexts.

By narrowing the problem space — “analyze diffs + apply internal guidelines” — I didn’t need a 175B parameter model. A smaller, fine-tuned model with access to the right data was faster, cheaper, and often more accurate.

In fact, the success of SLMs lies in being “just enough”:

• Just enough context (diff, not full repo).
• Just enough knowledge (wiki, not the internet).
• Just enough power (SLM, not massive LLM).

Lessons Learned

1. Context > Parameters Don’t throw a giant model at the problem. Give a smaller model the right context.
2. Create Knowledge Loops Your AI agent should learn from PR comments, wiki updates, and retrospectives. Knowledge shouldn’t get lost in Slack threads.
3. Scoping Is Everything Narrow your AI agent’s scope. Instead of “understand the repo,” aim for “review diffs against wiki.”
4. Humans + AI > AI Alone My manual comments didn’t go to waste. They seeded the agent with valuable insights, amplifying my effort across future PRs.

The Future of AI in Engineering Teams

What excites me is how this approach scales. Imagine every engineering team having an AI apprentice:

• Learns from PRs and wiki updates.
• Knows the team’s coding voice and evolving conventions.
• Provides instant, context-aware feedback.
• Reduces repetitive review cycles and lets humans focus on creative problem solving.

This isn’t a dream. With SLMs tied to structured knowledge sources, it’s already happening.

Closing Thought

The race isn’t about who has the biggest model. It’s about who can deploy AI effectively in the real world. For most engineering teams, success comes not from massive LLMs but from practical, scoped, Small Language Models that integrate into daily workflows.

So the next time you catch yourself thinking, “Didn’t I already review this exact thing last week?”, maybe it’s time to ask: 👉 “Can I teach my AI agent to do it for me?”

I’ve shared this story not just as a technical case study, but as a reminder that engineering is as much about people as it is about code. Every review comment, every wiki note, every shared insight is a small piece of our collective knowledge. By teaching our AI agents to respect and reuse that knowledge, we’re amplifying our teamwork and freeing ourselves to focus on what really matters: building meaningful systems.

If this perspective resonated with you, I’d love to hear your thoughts. Have you tried integrating AI into your code reviews or workflows? What worked, what didn’t?

🙏 Thanks for reading all the way through. If you’d like to follow along as I share more stories about real-world LLM implementations, Small Language Models, and practical AI engineering, hit that Follow button here on Medium. Let’s keep learning and building together.

A message from our Founder

Hey, Sunil here. I wanted to take a moment to thank you for reading until the end and for being a part of this community.

Did you know that our team run these publications as a volunteer effort to over 3.5m monthly readers? We don’t receive any funding, we do this to support the community. ❤️

If you want to show some love, please take a moment to follow me on LinkedIn, TikTok, **Instagram. You can also subscribe to our [weekly newsletter](https://newsletter.plainenglish.io/)**.

And before you go, don’t forget to clap and follow the writer️!