Running Local LLMs with OutSystems: A Practical Guide for Building Private, Flexible AI-Powered Apps

Large Language Models have become part of the normal development workflow. We use ChatGPT, Claude, GitHub Copilot, Amazon Bedrock, Azure OpenAI, and other hosted AI services to write code, summarize documents, generate tests, classify text, and build AI-powered business applications.

For most teams, hosted models are the easiest place to start. They are powerful, fast, scalable, and available with almost no infrastructure setup. But they also introduce trade-offs: data leaves your environment, cost grows with usage, model behavior can change when the provider updates the model, and your application depends on a third-party AI service being available.

Local LLMs give us another option.

Instead of sending requests to an external AI provider, we run the model ourselves on our own machine, server, VM, or controlled infrastructure. This changes the architecture, the cost model, the security model, and the performance profile.

It also changes how we think about model selection. In many real-world applications, the best answer is not always to use the biggest generic model available. A smaller model optimized for a specific activity, such as classification, summarization, extraction, coding assistance, or document analysis, can be faster, cheaper, easier to control, and more successful for that specific task.

This article explains how local LLMs work, how to configure models with Ollama, and how to connect a locally hosted model to ODC applications.

The goal is not to claim that local LLMs are always better. They are not. The goal is to understand when they are useful, how to integrate them properly, and how to avoid treating a demo architecture as a production architecture.

Before jumping into the implementation, we need to understand the architectural difference between a hosted LLM and a local LLM.

Most developers are already familiar with the hosted model approach. This is the architecture behind tools like ChatGPT, Claude, Gemini, GitHub Copilot, Claude Code, and many AI features exposed through cloud providers.

Hosted LLM architecture

In a hosted LLM architecture, the model does not run inside your application.

A user interacts with a UI, agent, developer tool, or business application. That client sends a request to an external AI provider such as OpenAI, Anthropic, Azure OpenAI, Amazon Bedrock, Google, or another managed platform. The provider runs the model in its own infrastructure and returns the generated response.

The architecture usually looks like this:

The important point is that the UI and the model are separate things.

When you use ChatGPT, Claude Code, Copilot, or an AI feature inside another product, you are usually interacting with a client interface. The actual model runs somewhere else, usually inside the provider’s data center. Your prompt, context, files, code snippets, or application data are sent to that provider, processed by the model, and then returned as a response.

That model is convenient because there is almost no infrastructure to manage. You get access to powerful models immediately, the provider handles scaling, and you can start building quickly.

But this convenience comes with trade-offs:

• Your data leaves your environment
• You pay through subscriptions, token usage, or platform consumption
• Your application depends on the provider’s availability
• You have limited control over model behavior and updates
• Model changes can affect consistency between releases

Local LLM architecture

Local LLMs change this architecture completely.

Instead of sending every prompt to an external provider, the model runs on infrastructure you control.

In the simplest setup, that infrastructure is your own computer.

The architecture looks like this:

In this local architecture, the model runs on infrastructure we control instead of inside a provider’s data center.

For this example, I am using Ollama to run the model locally.

Ollama is simply the runtime that loads the downloaded model and makes it available to other tools on the machine.

The user can still interact through familiar interfaces, such as Claude Code, Codex-style tools, Open WebUI, the Ollama terminal interface, or a custom application. The difference is where the request goes.

Instead of sending the prompt to a hosted AI provider, the client sends the request to the local runtime. The runtime then passes it to the model running on your machine, VM, or internal server.

The important distinction is this:

The UI is not the model.

Claude Code, Codex-style tools, chat interfaces, and custom applications are clients. They provide the interaction layer. The local runtime executes the model. The downloaded LLM provides the actual intelligence.

With this architecture, your machine becomes the inference environment. CPU, GPU, RAM, and VRAM directly affect the quality and speed of the experience. If the model is small and optimized, responses can feel fast. If the model is large or loaded on insufficient hardware, responses can be slow, sometimes painfully slow.

This is the main trade-off with local LLMs: you gain control, but you also inherit the infrastructure responsibility.

The advantages are clear:

• Your prompts and data can stay inside your own environment
• You are not charged per token by an external model provider
• You can experiment freely with different models
• You can work offline once the model is downloaded
• You control the model version being used
• You reduce dependency on hosted AI providers

But there are also disadvantages:

• You need capable hardware
• Larger models may be too slow or impossible to run locally
• Local models are usually weaker than the best hosted frontier models
• Setup and maintenance become your responsibility
• Performance depends heavily on model size and quantization
• Scaling beyond one machine requires additional architecture

This is why local LLMs should not be described as simply “better” or “worse” than hosted LLMs. They solve a different problem.

Hosted LLMs optimize for convenience, scale, and access to very strong models.

Local LLMs optimize for control, privacy, cost predictability, experimentation, and independence.

For OutSystems developers, this becomes especially interesting because OutSystems applications do not care whether the model is hosted by OpenAI, Azure, Amazon, Anthropic, or running locally behind an API.

What matters is whether the application can reach a compatible endpoint, send a request, receive a response, and safely use that response inside the business process.

Now that the difference between hosted and local LLMs is clear, we can move into the practical setup.

The goal is simple: run an LLM locally, expose it through an API, and then connect that API to OutSystems.

For this example, I will use Ollama because it gives us a fast way to download, run, and serve local models without manually configuring the full inference stack.

Step 1: Install Ollama

The first step is to install Ollama on the machine that will run the model.

Ollama is available for macOS, Linux, and Windows. You can download it from the official Ollama website:

https://ollama.com/download

After installation, Ollama runs as a local service on your machine. This is important because the model will not be called directly by OutSystems yet. First, the model needs to be available locally, and Ollama is what gives us that local runtime.

At this stage, the architecture is still simple:

Your machine
  -> Ollama
  -> Local model

Nothing is connected to OutSystems yet. We are only preparing the local LLM environment.

Step 2: Choose and install a model

After Ollama is installed, the next step is choosing which model you want to run locally.

You can browse the available models in the Ollama model library:

https://ollama.com/search

For the first test, I recommend starting with a smaller model. Smaller models are easier to download, faster to load, and more likely to run well on normal developer hardware.

Later in another article, we will look at how to choose the right model for each use case, including model size, training type, and quantization. For now, the goal is only to get a local model running.

For this example, I will use this model:

qwen3:8b-instruct-q4_K_M

This model name already tells us a few important things:

qwen3        -> model family
8b           -> model size, around 8 billion parameters
instruct     -> trained to follow instructions
q4_K_M       -> quantized version optimized for local execution

Do not worry if these terms are not fully clear yet. We will break them down later. At this stage, the important point is that this is a relatively practical model for a local test.

You can download the model using the command line:

ollama pull qwen3:8b-instruct-q4_K_M

This command downloads the model to your machine.

After the download finishes, you can run it with:

ollama run qwen3:8b-instruct-q4_K_M

At this point, you can interact with the model directly from the terminal.

For example, ask:

Hello, who are you?

If the model responds, the local runtime is working.

At this stage, we have not connected anything to OutSystems yet. We have only confirmed that the model can run locally on our machine. That is the foundation we need before exposing it through an API and using it from an OutSystems application.

Step 3: Understand what is happening locally

When you run this command:

ollama run qwen3:8b-instruct-q4_K_M

Ollama loads the model and performs inference on your machine.

That means your local hardware is doing the work. Your CPU, GPU, RAM, and VRAM directly affect the speed of the response. A small optimized model can respond quickly. A larger model can be much slower, especially if your machine does not have enough GPU memory.

This is one of the most important differences from hosted LLMs.

With hosted models, the provider owns the infrastructure.

With local models, you own the infrastructure.

That gives you more control, but it also means performance is now your responsibility.

Step 4: Use the model from another interface

The terminal is useful for testing, but it is not the only way to interact with the model.

Once the model is running locally, other tools can connect to it. For example:

• Ollama’s own interface
• Open WebUI
• Claude Code-style developer tools
• Codex-style tools
• Custom applications
• REST clients
• Backend services

This is where the architecture starts to become more useful.

The user does not need to interact with the model directly. A UI, tool, or application can send requests to the local runtime, and the runtime sends those requests to the model.

Conceptually, the flow becomes:

User
  -> UI or application
  -> Ollama local runtime
  -> Local model
  -> Response

This is the same idea we will later use with OutSystems. The OutSystems application does not need to know how the model is loaded into memory. It only needs to call an API.

Step 5: Confirm the local API

Ollama exposes a local HTTP API that tools on the same machine can call.

If you want other machines on the network to reach it, you need to enable network access or configure the host binding.

The common local address is:

http://localhost:11434

This means that tools running on the same machine can send HTTP requests to Ollama.

For example, Ollama’s native API can be called through endpoints under this local server.

A simple local test can be done with:

curl http://localhost:11434/api/tags

This returns the models currently available in your local Ollama environment.

If this works, it means Ollama is running and exposing its local API correctly.

Step 6: Enable access from the network

By default, a local service is usually only accessible from the same machine.

That is fine when you are using the terminal or a local UI, but it is not enough when another system needs to call the model.

This matters for OutSystems.

An OutSystems application running in ODC is not running on your laptop. It runs in the OutSystems cloud environment. So, from the OutSystems application’s perspective, this address will not work:

http://localhost:11434

Why?

Because localhost would refer to the OutSystems runtime environment, not your computer.

To allow another system to call your local model, you need to expose the Ollama API through a reachable address.

For local development and demos, this can be done with a reverse proxy or tunnel such as ngrok. For production, this should be done through a controlled API gateway, private network, VPN, or internal reverse proxy.

We will cover that part later, but before exposing anything externally, there is one more important detail.

Step 7: Use the OpenAI-compatible API

Ollama can expose endpoints compatible with parts of the OpenAI API.

This is extremely useful because many AI tools and platforms already know how to talk to OpenAI-style APIs. Instead of building a completely custom integration, you can often configure the local model as if it were an OpenAI-compatible model endpoint.

The common OpenAI-compatible base URL is:

http://localhost:11434/v1

For chat completions, the endpoint is:

POST http://localhost:11434/v1/chat/completions

A request looks similar to this:

{
  "model": "qwen3:8b-instruct-q4_K_M",
  "messages": [
    {
      "role": "system",
      "content": "You are a practical technical assistant."
    },
    {
      "role": "user",
      "content": "Explain what a local LLM is in one paragraph."
    }
  ],
  "temperature": 0.2
}

This compatibility is the bridge that makes the OutSystems integration much simpler.

OutSystems expects AI models to be exposed through provider-style APIs. In ODC, you can configure AI models from supported providers, and for private or custom models, you can connect to a custom endpoint.

Luckily for us, Ollama can expose the local model through an OpenAI-compatible API shape.

That means the local model can be treated less like a special local experiment and more like a private AI endpoint that OutSystems can call.

The full development flow becomes:

Install Ollama
  -> Download a model
  -> Run the model locally
  -> Confirm the local API
  -> Enable network access
  -> Use the OpenAI-compatible endpoint
  -> Configure it in OutSystems

At this point, the local model is no longer just something we can use from the terminal. It becomes an API-backed model that can be integrated into an OutSystems application.

At this point, we have the local architecture working.

The model is installed on our machine, Ollama can run it locally, and we can interact with it using the terminal or another local interface.

The current architecture looks like this:

This works well when the user and the model are on the same machine. But this is not enough for an OutSystems application.

An OutSystems app running in ODC does not run on your laptop. It runs on the OutSystems cloud runtime. That means it cannot call your local machine using:

http://localhost:11434

From the OutSystems server’s perspective, localhost means the OutSystems runtime itself, not your computer.

So if we want an OutSystems application to call our local model, we need to expose the local Ollama API through a URL that the OutSystems server can reach.

For a demo or development environment, one simple way to do this is with a reverse proxy tunnel such as ngrok.

The architecture then becomes:

In this setup, ngrok creates a public HTTPS endpoint and forwards requests to Ollama running locally on our machine.

The flow becomes:

User
  -> OutSystems app in the browser
  -> OutSystems server runtime
  -> ngrok public HTTPS endpoint
  -> Ollama running locally
  -> Local LLM model

This gives OutSystems a reachable API endpoint while still allowing the model itself to run on infrastructure we control.

For this example, that infrastructure can be your own computer. In a more realistic enterprise setup, it could be a cloud VM, an internal server, or a machine inside your company network.

The important concept is the same: OutSystems does not need to know that the model is running locally. It only needs a reachable API endpoint that follows the expected contract.

Before we configure OutSystems, let’s expose the local Ollama API using ngrok.

Step 8: Expose the local API with ngrok

ngrok is a tunneling tool that can expose a local service through a public HTTPS URL.

In our case, Ollama is running locally on port 11434, so we need ngrok to forward external traffic to that port.

You can download ngrok from the official website:

https://ngrok.com/download

After installing ngrok and signing in, you can start a tunnel in the CLI with:

This is acceptable for a demo, but do not expose an LLM endpoint publicly without authentication and proper controls.

ngrok http 11434

This creates a public HTTPS URL that forwards traffic to:

http://localhost:11434

However, exposing an LLM endpoint without protection is a bad idea, even for a demo.

At minimum, add basic authentication.

For example:

ngrok http 11434 --basic-auth="User:Pass" --host-header="localhost:11434"

Breaking it down for better clarity we have:

ngrok http 11434

exposes the local Ollama server.

--basic-auth="User:Pass"

adds basic authentication to the public endpoint.

--host-header="localhost:11434"

ensures the forwarded request uses the expected local host header.

After running the command, ngrok will display a forwarding URL similar to:

https://abc123.ngrok-free.app

This URL now points to your local Ollama server.

So the local address:

http://localhost:11434

becomes reachable externally through something like:

https://abc123.ngrok-free.app

This is the URL we can use from OutSystems.

Step 9: Use the OpenAI-compatible endpoint

Now we need one more detail.

OutSystems AI model configuration expects the model to be available through an API-style contract. Luckily, Ollama provides compatibility with parts of the OpenAI API, which means existing tools and platforms can connect to Ollama using OpenAI-style endpoints.

For Ollama, the OpenAI-compatible base path is:

http://localhost:11434/v1

After exposing Ollama through ngrok, that becomes:

https://abc123.ngrok-free.app/v1

The chat completions endpoint becomes:

POST https://abc123.ngrok-free.app/v1/chat/completions

This is the key part.

We are not asking OutSystems to understand Ollama specifically. We are exposing the local model through an API shape that is already familiar to AI platforms and tools.

Conceptually, OutSystems will call:

https://abc123.ngrok-free.app/v1

ngrok will forward the request to:

http://localhost:11434/v1

and Ollama will route the request to the selected local model.

The resulting architecture is:

OutSystems
  -> ngrok HTTPS endpoint
  -> Ollama OpenAI-compatible API
  -> Local model

This is what makes the integration practical.

The local model is still running on infrastructure we control, but from the OutSystems side it behaves like a remote AI model endpoint.

Step 10: Configure the local model in ODC

At this point, the model is running locally, and ngrok is exposing the Ollama API through a public HTTPS endpoint.

Now we can configure that endpoint in OutSystems Developer Cloud.

In the ODC Portal, go to:

Integrate -> AI models

Then add a new model endpoint.

The configuration should include:

Name: Any name
Model ID: qwen3:8b-instruct-q4_K_M (Your model name)
URL: https://abc123.ngrok-free.app/v1 (Ngrok link)

It will look like this:

The exact URL will be the forwarding URL generated by ngrok, followed by /v1.

For example, if ngrok gives you this URL:

https://abc123.ngrok-free.app

then the OpenAI-compatible base URL should be:

https://abc123.ngrok-free.app/v1

ODC will use this base URL and call the chat completions endpoint from there.

Conceptually, the request becomes:

ODC
  -> https://abc123.ngrok-free.app/v1/chat/completions
  -> ngrok
  -> http://localhost:11434/v1/chat/completions
  -> Ollama
  -> local model

This is why the /v1 path matters. It points OutSystems to the OpenAI-compatible API exposed by Ollama.

Configuring authentication

If you exposed the endpoint with Basic Auth in ngrok, you also need to add an authorization header in ODC.

For example, if you started ngrok like this:

ngrok http 11434 --basic-auth="User:Pass" --host-header="localhost:11434"

then the credentials are:

User:Pass

Basic Auth expects those credentials to be encoded in Base64 and prefixed with Basic.

You can encode the value using a Base64 tool such as:

https://www.base64encode.org/

For example:

User:Pass

becomes something like:

VXNlcjpQYXNz

Then in the ODC model endpoint configuration, add this header:

Header name: Authorization
Header value: Basic VXNlcjpQYXNz

The final value must include the word Basic, followed by a space, followed by the Base64 encoded credentials.

Test the endpoint

After entering the model name, model ID, URL, and authorization header, use the ODC test option to validate the endpoint.

If the test succeeds, the model is now available inside ODC.

From this point forward, you can reference it from your OutSystems ODC agents like any other configured AI model.

That is the key benefit of exposing Ollama through an OpenAI-compatible API.

OutSystems does not need to know that the model is running on your computer, VM, or internal server. From the application side, it is just another model endpoint.

The development architecture is now:

User
  -> OutSystems app
  -> ODC agent
  -> configured AI model endpoint
  -> ngrok
  -> Ollama
  -> local LLM

For a demo or proof of concept, this is enough to prove the integration works.

For production, I would replace ngrok with a controlled network path, such as an API gateway, private endpoint, VPN, internal reverse proxy, or cloud-hosted inference service. The same concept still applies, but the endpoint should be secured and managed like any other enterprise integration.

Hopefully this helps you save a few tokens. 🙂

Happy coding.