From Prompts to Policies: Getting Started with Prompt Engineering Policies in WSO2 AI Gateway

Large Language Models (LLMs) are powerful but without control, they are unpredictable. As organizations start integrating AI into real systems, the challenge shifts from “how do we call an LLM?” to “how do we control what the LLM does?”

This is where the WSO2 AI Gateway becomes essential.

With the AI Gateway, developers can expose AI endpoints in a governed, secure, and consistent way. More importantly, it introduces Prompt Engineering Policies a way to centrally control and standardize LLM behavior without changing application code.

In this blog, we’ll explore WSO2 AI Workspace & how to:

• Configure LLM providers
• Set up the AI Gateway locally
• Deploy the LLM Provider to the AI Gateway
• Apply Prompt Decorators and Prompt Templates

Prerequisites

• You’ll need an API key from your preferred LLM provider. In this blog, I will be using the Mistral API.

• Set up WSO2 AI Workspace by visiting console.bijira.dev and creating an organization. Then navigate to ai-workspace.bijira.dev and sign in with your account.

Step 1: Configure a LLM Provider

When we log in to the AI Workspace, we are presented with a user-friendly quick start guide that clearly walks us through the setup process, allowing us to smoothly follow the steps and complete the configuration.

Navigate to LLM Providers and click Add LLM Service Provider. Then select your preferred provider (e.g., OpenAI, Mistral, etc.). Fill in the required details:

API Key: Paste the API key you obtained from your LLM provider along with any other required credentials

Once all details are filled in, click Add Provider. After successful configuration, the provider will appear in the providers list and will be available for use in your AI Gateway setup.

Step 2: Set Up the AI Gateway

Navigate to AI Gateways and click + Add AI Gateway. Fill in the required gateway details and proceed.

After creation, click View Gateway Configurations. This section provides all the necessary keys and commands required to run the gateway locally. Then, you can simply follow the provided instructions and commands step by step as follows.

1: Download the Gateway

Run the following command in your terminal to download and extract the AI Gateway:

curl -sLO https://github.com/wso2/api-platform/releases/download/ai-gateway/v1.0.0/wso2apip-ai-gateway-1.0.0.zip && \
unzip wso2apip-ai-gateway-1.0.0.zip

2: Configure the Gateway

Create the environment configuration file with the required variables:

cat > wso2apip-ai-gateway-1.0.0/configs/keys.env << 'ENVFILE'
MOESIF_KEY=<your-moesif-key>
GATEWAY_CONTROLPLANE_HOST=connect.bijira.dev
GATEWAY_REGISTRATION_TOKEN=<your-gateway-token>
ENVFILE

3: Start the Gateway

Navigate to the gateway directory:

cd wso2apip-ai-gateway-1.0.0

Start the gateway using Docker:

docker compose --env-file configs/keys.env up

Once these steps are completed, the AI Gateway will start running locally.

Step 3: Deploy the LLM Provider to the AI Gateway

With the gateway running, return to your configured LLM provider in AI Workspace and deploy it to the gateway. Once deployed, the dashboard will display key elements,

API Key Generation: Use this interface to create API keys for your applications. These keys can be rotated independently of the underlying LLM provider key.

Navigate to the deployed API and select the Chat Completions resource. Use the built-in interface to generate a sample request. This will include the correct Gateway URL, headers, and payload. You can then copy the generated cURL command and run it locally. Sample command:

curl -X POST 'https://localhost:8443/mistral/v1/chat/completions' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -H 'X-API-Key: <YOUR_API_KEY>' \
  -d '{
    "model": "mistral-large-latest",
    "messages": [
      {
        "role": "user",
        "content": "How far is the moon from earth?"
      }
    ]
  }'

Now we can apply guardrails/policies to the LLM provider as needed.

Step 4: Apply Prompt Decorator policy

The Prompt Decorator policy allows you to modify prompts before they are sent to the upstream LLM by either prepending or appending additional instructions. This is useful for enforcing consistent tone, format, and response constraints across all AI interactions.

How to Apply a Prompt Decorator

1. Navigate to your deployed LLM proxy in AI Workspace
2. Go to the Guardrails section
3. Select Prompt Decorator and click Add
4. Provide the required configuration

Configuration Example:

[{"role": "system", "content": "Respond in bullet points. Keep answers under 100 words."}]

In advance settings,

• When append = true, the decoration is added after the existing prompt
• When append = false (default), the decoration is prepended before the prompt

In most cases, prepending is preferred to ensure the instruction strongly influences the model’s response.

After adding the policy:

• Save the configuration
• Redeploy the provider for the changes to take effect

Once applied, every request passing through the gateway will automatically include the decorator instructions ensuring consistent and controlled LLM behavior without modifying client applications.

This is the results I got for the same request I used earlier,

Step 5: Apply Prompt Template Policy

The Prompt Template policy allows you to define reusable, parameterized prompts in the AI Gateway. Instead of sending raw prompts from the client application, users reference a template, and the gateway dynamically builds the final prompt before sending it to the LLM. Use same steps as Prompt Decorator and configure this,

In addition there are advanced configurations,

• On Missing Template: Defines behavior when a referenced template is not found (error or passthrough).
• On Unresolved Placeholder: Defines how missing parameters are handled (keep, empty, or error).

We will use following example template, and try the same request. Make sure to use the given name in request,

name: new-temp
Prompt: Translate the following text from [[source_language]] to [[target_language]]:[[text]]

Save and deploy the provider and then invoke,

💡 Real Value in Practice

With prompt engineering policies in place, developers can:

• Build AI-powered applications faster without worrying about prompt design complexity
• Maintain consistent output quality across different use cases
• Dynamically adjust AI behavior (tone, format, structure) without code changes
• Scale AI usage safely in enterprise environments

🎯 Final Takeaway

Prompt Engineering Policies transform the AI Gateway from a simple routing layer into a powerful AI control plane. They allow organizations to treat prompts as governed, reusable assets, enabling scalable, secure, and maintainable AI integration.

In short: You don’t just call the LLM, you control how it thinks before it responds.