Run Your Own In-House AI Knowledge Assistant with AnythingLLM + Ollama

In most companies, project knowledge is scattered across documents, wiki pages, design diagrams, and training videos. When a new developer joins, they often spend weeks asking around or digging through old docs to find relevant information. What if you could centralize all that knowledge into a private AI assistant that understands your project — and runs completely in-house without sending data outside?

That’s exactly what I built using AnythingLLM and Ollama, and in this post, I’ll walk you through the setup and benefits.

Why In-House AI?

Most popular AI tools are cloud-based, meaning your queries and data go outside your organization. For sensitive projects, this is a big concern. By running AI locally:

• ✅ Full data privacy — nothing leaves your machine or company network.
• ✅ Custom knowledge — connect your own docs, wiki, and videos.
• ✅ No subscriptions — run on your own hardware, no per-user billing.

Tools Used

1. Ollama - A tool that lets you run large language models (LLMs) locally on your laptop or server. You can download models like llama3, mistral, codellama, and nonmic-text and run them without needing internet or a paid API.

1. AnythingLLM — an open-source chat interface where you can upload documents, connect sources, and query via your chosen LLM.

https://github.com/Mintplex-Labs/anything-llm

Setup Steps

Here’s how I set up my project assistant:

1. Install Ollama

Download and install Ollama for your OS. Pull a model you want to use:

ollama pull llama3.1:latest

(You can try other models like codellama:latest, llama3, etc.)

2. Install AnythingLLM

Clone the repo and run it with npm:

git clone https://github.com/Mintplex-Labs/anything-llm.git
cd anything-llm
npm install
npm run dev

3. Connect AnythingLLM to Ollama

In the settings, choose Ollama as the LLM provider and configure it with your local URL (usually [http://localhost:11434).](http://localhost:11434).)

4. Load Your Knowledge Sources

I uploaded:

• Project wiki links
• API specifications
• KT (Knowledge Transfer) videos
• Design diagrams

AnythingLLM processes these into embeddings so the AI can retrieve relevant context when answering.

5. Start Chatting With Your Project

Now, I can ask things like:

• “What APIs handle user authentication?”
• “Summarize the system design diagram for me.”
• “Explain the KT video on deployment in 5 points.”
• “What is the Curl of my following API.

The AI answers directly from my project’s data — fast and context-aware.

Benefits I’ve Seen

• 🔒 Private: All data stays within my laptop/network.
• ⚡ Faster onboarding: New team members can self-serve answers instead of asking leads repeatedly.
• 📚 Centralized knowledge: No more scattered documents.
• 💸 Cost-effective: No subscription fees, everything runs locally.

Bringing It All Together

With AnythingLLM + Ollama, I’ve basically built an AI-powered project knowledge base that’s private, secure, and highly useful for day-to-day development.

If your team struggles with scattered documentation or knowledge sharing, try this setup. You’ll be surprised how quickly it turns into your go-to project assistant.

Happy Learning!

• — Rajat Sharma*