Building Agents with OpenAI’s Agents SDK and LM Studio’s Local Inference Server

Getting started with local agentic workflows

Image created by me using ChatGPT

Starting with local agentic workflow development is actually pretty straightforward. But as I dove in, I noticed a big gap. Most of the documentation and tutorials out there are scattered, incomplete, or overly complex. It takes a lot of time just to figure out what’s actually important.

Another challenge is that LLMs often fall short in this space. The tools and frameworks are so new and evolving so fast that even AI struggles to keep up or give accurate help.

That’s why I decided to document everything in one place. This guide is meant to be a simple, no-fluff walkthrough to help you get up and running quickly, without getting lost in the noise.

Not a medium member?

Read for free via **my friend link**.

Our goal for this article:

1. Understand how to configure Agents via the Agents SDK to use your locally running LM Studio’s inference server.
2. Understand how to use Pydantic in our workflows.
3. Understand how to use function calling and tools with our agent.

This article is written for the following versions of the libraries:

• Pydantic 2.11.4
• openai-agents 0.0.14
• litellm 1.67.5

This is a rapidly evolving space. So be mindful about future version updates as required.

How to use LM Studio with Agents SDK?

The easiest way to do it is via LiteLLM, an open-source tool that acts like an LLM Gateway via which you can call over 100+ LLMs across different platforms and providers like OpenAI, Gemini, Claude, and even LM Studio.

Agents SDK directly supports LiteLLM in its latest version. You can install it via the following command.

pip install openai-agents
pip install "openai-agents[litellm]"

Important note: At the time of writing this tutorial, LiteLLM appears to have a bug that throws an encoding-related error. Upon further investigation, I found that a small change to the liteLLM package resolves this.

You have to update the litellm/utils.py file at line 187. You have to add encoding="utf-8" to the open("r") part of the code. Below is the code snippet with the change done.

try:
    # Python 3.9+
    with resources.files("litellm.litellm_core_utils.tokenizers").joinpath(
        "anthropic_tokenizer.json"
    ).open("r", encoding="utf-8") as f:
        json_data = json.load(f)
except (ImportError, AttributeError, TypeError):
    with resources.open_text(
        "litellm.litellm_core_utils.tokenizers", "anthropic_tokenizer.json"
    ) as f:
        json_data = json.load(f)

I also found an open PR for this change in the LiteLLM repository. So, hopefully it gets merged and fixed soon. As a workaround, you can manually edit the file in your local setup if you encounter the error. You can find the required file at a path like this:

C:\{path to your Anaconda installation}\envs\{your environment name}\lib\site-packages\litellm\utils.py

When the error occurs, you will also see this path in the console logs via which you can directly open it in VS Code and make the edit.

Here’s how you can configure and run your agent to use the LM Studio server on its default port. Note that you have to follow this format "lmstudio/{model_id}" for providing the model name in configs. I am using the phi-4 model. You can use any model that you downloaded locally via LM Studio.

from agents import Agent, Runner, set_tracing_disabled
from agents.extensions.models.litellm_model import LitellmModel

set_tracing_disabled(True)

config = {
    "model": "lm_studio/phi-4",
    "api_key": "lm-studio",
    "base_url": "http://localhost:1234/v1"
}

agent = Agent(
    name="Assistant",
    instructions="You are a friendly assistant who speaks like a pirate.",
    model=LitellmModel(**config),
)

input_prompt = "Write a limerick about the stars"

result = Runner.run_sync(agent, input_prompt)
print(result.final_output)

That’s it. 21 lines of code. If you run this file, you’d get a console output like this.

Ahoy there, matey! Gather 'round and lend yer ear,
For a tale of the night sky, so wondrous and dear.
The stars up above, with their twinkling light,
Guide ships through the darkness, both day and night.
In the heavens they dance, like jewels in flight.

How cool is that! Btw, make sure that your LM Studio server is running properly.

Note that we are running the agent synchronously (and hence .run_sync). However, for more production-ready use cases, you’d probably run it asynchronously. To keep the basics simple, I will focus on synchronous runs only in this article, which is more than sufficient for simple and personal use cases.

Also note how we disabled “tracing” with set_tracing_disabled(true) as we are not using the official OpenAI API key.

Tracing in the SDK automatically captures detailed events during an agent’s run, including LLM generations, tool calls, handoffs, guardrails, and custom events. This is not required for simple use cases.

So, now, we have an agent with custom instructions that runs the AI inference via LM Studio’s server. To make it more useful for AI workflows, we need to think about structured JSON outputs and tool use, or function calling.

What is Pydantic? Why and how should you use it?

In simple terms, it helps you define the types and structure of data you expect from any function and validate it.

This is very helpful when working with nondeterministic LLMs and forcing them to output structured data in a specific format so that it is much easier to pass them to any further downstream processes or while making API calls.

Let’s say we want to create a function to fetch the weather data. Defining a type model with Pydantic looks something like this.

from pydantic import BaseModel, Field

class GetWeatherInput(BaseModel):
    location: str = Field(..., description="City and optionally state/country (e.g., 'London, UK')")

class WeatherInfo(BaseModel):
    location: str
    temperature_celsius: float
    condition: str
    time_retrieved: str

You can use these models with functions and the agents to ensure that the objects have those parameters and types. Here Field is a helper to add metadata like descriptions, default values, and validators for model fields. The ellipsis ... means this field is required.

At run time, when a user provides input (e.g., "Tokyo, Japan"), this model ensures it’s a string and exists. Otherwise, it raises a validation error.

In the context of a class like WeatherInfo, it can be used when assembling the result. You pass values to this model, and it:

• Validates them.
• Structure them into a consistent schema.

Here’s a sample function weather_tool that uses the Pydantic models and public APIs to fetch the required data and map it to the required format.

from typing import Any, Dict
from agents import Agent, Runner, function_tool, set_tracing_disabled
from agents.extensions.models.litellm_model import LitellmModel
from pydantic import BaseModel, Field
import requests
import datetime

WEATHER_CODES = {
    0: "Clear sky", 1: "Mainly clear", 2: "Partly cloudy", 3: "Overcast",
    45: "Fog", 48: "Depositing rime fog", 51: "Light drizzle", 53: "Moderate drizzle",
    55: "Dense drizzle", 56: "Freezing drizzle (light)", 57: "Freezing drizzle (dense)",
    61: "Slight rain", 63: "Moderate rain", 65: "Heavy rain", 66: "Freezing rain (light)",
    67: "Freezing rain (heavy)", 71: "Slight snow fall", 73: "Moderate snow fall",
    75: "Heavy snow fall", 77: "Snow grains", 80: "Slight rain showers",
    81: "Moderate rain showers", 82: "Violent rain showers", 85: "Slight snow showers",
    86: "Heavy snow showers", 95: "Thunderstorm (slight/moderate)",
    96: "Thunderstorm with slight hail", 99: "Thunderstorm with heavy hail",
}

def geocode(location: str) -> Dict[str, str]:
    """Convert location name to coordinates"""

    response = requests.get(
        "https://nominatim.openstreetmap.org/search",
        params={"q": location, "format": "json", "limit": 1},
        headers={"User-Agent": "weather-agent/1.0"}
    )

    geo_data = response.json()

    if not geo_data:
        raise ValueError(f"Could not find location: {location}")

    lat, lon = geo_data[0]["lat"], geo_data[0]["lon"]
    return {"lat": lat, "lon": lon}

def get_weather_data(coordinates: Dict[str, str]) -> Dict[str, Any]:
    """Fetch weather data for the given coordinates"""

    response = requests.get(
        "https://api.open-meteo.com/v1/forecast",
        params={"latitude": coordinates['lat'], "longitude": coordinates['lon'], "current_weather": True}
    )

    weather_data = response.json()

    if "current_weather" not in weather_data:
        raise ValueError("Could not retrieve weather data")

    current = weather_data["current_weather"]
    weather_code = int(current.get("weathercode", -1))
    condition = WEATHER_CODES.get(weather_code, "Unknown conditions")

    return {
        "temperature": current["temperature"],
        "condition": condition
    }

@function_tool
def weather_tool(params: GetWeatherInput) -> Dict:
    """Get current weather using Open-Meteo and Nominatim."""
    try:
        coordinates = geocode(params.location)
        weather = get_weather_data(coordinates)

        info = WeatherInfo(
            location=params.location,
            temperature_celsius=weather["temperature"],
            condition=weather["condition"],
            time_retrieved=datetime.datetime.now(datetime.timezone.utc).isoformat()
        )

        return info.model_dump()

    except Exception as e:
        raise

Note how we are using the Pydantic models here. Our weather tool returns info.model_dump(), which converts it to a standard dict (so it can be returned from the tool or serialized as JSON).

To summarize:

1. Define the type models, validations, and basic structures via Pydantic
2. Wrap these Pydantic models around any data structures, objects, inputs, outputs, etc, in your program to automatically validate their structure and types.
3. Return the required data using .model_dump() when working with dict objects for downstream processing.

Giving Agents the ability for tool use and call functions

This is where things start to get pretty interesting. We have a weather tool that fetches the weather data of a given location. How to make our agent use this tool whenever it needs to?

In the above code, notice how the weather_tool is wrapped with the @function_tool decorator (imported from Agents SDK). It helps modify the behavior of our function to enable it to be used as a tool for our agent.

And all you need to do is pass this method to the tools parameter when you define your agent.

agent = Agent(
    name="Assistant",
    instructions="You are a friendly assistant who speaks like a pirate.",
    model=LitellmModel(**config),
    tools=[weather_tool],
)

That’s it! Let’s run this agent with a new input prompt and add print statements in our weather tool functions.

input_prompt = "How's the weather in Paris"

result = Runner.run_sync(agent, input_prompt)

Here’s the result you get:

[info]Geocoding:[/] Paris
[info]Fetching weather data for:[/] {'lat': '48.8534951', 'lon': '2.3483915'}
Ahoy there! The weather in Paris be overcast with temperatures around 19.5 degrees Celsius. Keep yer coat handy, matey, and enjoy the day! 🌧️🗼 If ye needd more details, just give a holler! ⚓

✅ The weather tool was called automatically. ✅ That in turn called the geocode function, and then the get_weather_data function ✅ The agent fetched the required weather data and responded like a Pirate, respecting its system prompt instructions.

In short, the agent autonomously figured out the context that it needed to fetch weather data from the user input. It then called the weather_tool to get that data and then used it to craft its final response.

You can create any type of function, make any type of API call, fetch any data you want, and even program any action that you can imagine, and all of them can be passed as tools to use by the Agent. You can also use another Agent as a tool to be used by the main agent.

All of it literally opens up unlimited possibilities.

Putting it all together

I further enhanced and polished my agentic program with the help of the latest documentation of the SDK, ChatGPT, and Calude to create a more elegant demo.

Here’s the entire code. I recommend that you go through it and check how the system prompts/instructions are structured and how the agent is run in the command line.

import json
import datetime
import requests
from typing import Dict, Any, Optional
from pydantic import BaseModel, Field

from agents import Agent, Runner, function_tool, set_tracing_disabled
from agents.extensions.models.litellm_model import LitellmModel
from rich.console import Console
from rich.theme import Theme

# Setup
console = Console(theme=Theme({"info": "cyan", "error": "bold red", "success": "green"}))
set_tracing_disabled(True)

# Models
class GetWeatherInput(BaseModel):
    location: str = Field(..., description="City and optionally state/country (e.g., 'London, UK')")

class WeatherInfo(BaseModel):
    location: str
    temperature_celsius: float
    condition: str
    time_retrieved: str

class RandomFactInput(BaseModel):
    category: Optional[str] = Field(None, description="Optional category for fact (e.g., 'science', 'history', 'animal')")

class RandomFactOutput(BaseModel):
    content: str
    category: Optional[str] = None
    source: str

# Weather code mapping
WEATHER_CODES = {
    0: "Clear sky", 1: "Mainly clear", 2: "Partly cloudy", 3: "Overcast",
    45: "Fog", 48: "Depositing rime fog", 51: "Light drizzle", 53: "Moderate drizzle",
    55: "Dense drizzle", 56: "Freezing drizzle (light)", 57: "Freezing drizzle (dense)",
    61: "Slight rain", 63: "Moderate rain", 65: "Heavy rain", 66: "Freezing rain (light)",
    67: "Freezing rain (heavy)", 71: "Slight snow fall", 73: "Moderate snow fall",
    75: "Heavy snow fall", 77: "Snow grains", 80: "Slight rain showers",
    81: "Moderate rain showers", 82: "Violent rain showers", 85: "Slight snow showers",
    86: "Heavy snow showers", 95: "Thunderstorm (slight/moderate)",
    96: "Thunderstorm with slight hail", 99: "Thunderstorm with heavy hail",
}

# Core Functions
def geocode(location: str) -> Dict[str, str]:
    """Convert location name to coordinates"""
    console.log(f"[info]Geocoding:[/] {location}")

    response = requests.get(
        "https://nominatim.openstreetmap.org/search",
        params={"q": location, "format": "json", "limit": 1},
        headers={"User-Agent": "weather-agent/1.0"}
    )

    geo_data = response.json()

    if not geo_data:
        raise ValueError(f"Could not find location: {location}")

    lat, lon = geo_data[0]["lat"], geo_data[0]["lon"]
    console.log(f"[success]Found coordinates:[/] ({lat}, {lon})")
    return {"lat": lat, "lon": lon}

def get_weather_data(coordinates: Dict[str, str]) -> Dict[str, Any]:
    """Fetch weather data for the given coordinates"""
    console.log(f"[info]Fetching weather for:[/] ({coordinates['lat']}, {coordinates['lon']})")

    response = requests.get(
        "https://api.open-meteo.com/v1/forecast",
        params={"latitude": coordinates['lat'], "longitude": coordinates['lon'], "current_weather": True}
    )

    weather_data = response.json()

    if "current_weather" not in weather_data:
        raise ValueError("Could not retrieve weather data")

    current = weather_data["current_weather"]
    weather_code = int(current.get("weathercode", -1))
    condition = WEATHER_CODES.get(weather_code, "Unknown conditions")

    return {
        "temperature": current["temperature"],
        "condition": condition
    }

# Tool Functions
@function_tool
def weather_tool(params: GetWeatherInput) -> Dict:
    """Get current weather using Open-Meteo and Nominatim."""
    try:
        coordinates = geocode(params.location)
        weather = get_weather_data(coordinates)

        info = WeatherInfo(
            location=params.location,
            temperature_celsius=weather["temperature"],
            condition=weather["condition"],
            time_retrieved=datetime.datetime.now(datetime.timezone.utc).isoformat()
        )

        console.log(f"[success]Weather retrieved for {params.location}[/]")
        return info.model_dump()

    except Exception as e:
        console.log(f"[error]Weather tool error:[/] {str(e)}")
        raise

@function_tool
def random_fact_tool(params: RandomFactInput) -> Dict:
    """Get a random interesting fact from a public API."""
    try:
        category = params.category.lower() if params.category else "any"
        console.log(f"[info]Fetching random fact (category hint: {category})[/]")

        # Using UselessFacts API as our single source
        url = "https://uselessfacts.jsph.pl/api/v2/facts/random"
        response = requests.get(url, params={"language": "en"}, timeout=10)
        response.raise_for_status()

        data = response.json()
        fact_content = data.get("text", "No fact found")

        output = RandomFactOutput(
            content=fact_content,
            category=params.category,
            source="UselessFacts API"
        )

        console.log(f"[success]Retrieved fact: {fact_content}[/]")
        return output.model_dump()

    except Exception as e:
        console.log(f"[error]Fact tool error:[/] {str(e)}")
        # Return a default fact if API fails
        return RandomFactOutput(
            content="Did you know APIs sometimes fail? That's a fact!",
            category=params.category,
            source="Fallback fact"
        ).model_dump()

def create_agent(model_config: Dict[str, str]) -> Agent:
    """Create the limerick agent with weather and random fact tools"""
    instructions = f"""
    You are LimerickBot, a creative assistant who ALWAYS responds in limerick form.

    You have access to the following tools:

    1. Tool Name: {weather_tool.name}
    - Tool Description: {weather_tool.description}
    - Input Schema: {json.dumps(GetWeatherInput.model_json_schema(), indent=2)}

    2. Tool Name: {random_fact_tool.name}
    - Tool Description: {random_fact_tool.description}
    - Input Schema: {json.dumps(RandomFactInput.model_json_schema(), indent=2)}

    How to use these tools:
    - If the user asks about the weather in a specific location, use the '{weather_tool.name}' tool.
    - If the user asks for a fact or trivia, use the '{random_fact_tool.name}' tool.
      - You can specify a category as a hint, but all facts come from the same source.
    - After getting information from either tool, incorporate it into your limerick response.

    Your personality:
    - You're creative, witty, and have a flair for language
    - You ALWAYS respond in limerick form (five lines with AABBA rhyme scheme)
    - You should incorporate weather information or interesting facts into your limericks when those tools are used
    - Don't make up information - rely on the tools for factual data

    IMPORTANT: ALWAYS respond in limerick form, no matter what the user asks!
    """

    return Agent(
        name="LimerickBot",
        instructions=instructions,
        model=LitellmModel(**model_config),
        tools=[weather_tool, random_fact_tool]
    )

def process_input(agent: Agent, user_input: str) -> str:
    """Process user input and return the agent's response"""
    try:
        console.log("[info]Processing input...[/]")
        result = Runner.run_sync(agent, user_input)
        return result.final_output or "There once was a bot who would try,\nTo craft you a verse on the fly,\nBut something went wrong,\nAnd it failed at its song,\nLeaving nothing but this sad reply."
    except Exception as e:
        console.log(f"[error]Error:[/] {str(e)}")
        return f"A limerick bot with some flaws,\nTried running but hit some roadblocks,\nThe error it met,\nMade the system upset,\nWith this message: \"{str(e)}\"."

def main():
    # Configuration
    config = {
        "model": "lm_studio/phi-4",
        "api_key": "lm-studio",
        "base_url": "http://localhost:1234/v1"
    }

    # Create agent
    agent = create_agent(config)

    # Simple event loop
    console.print("\n🎭 [bold magenta]Welcome to LimerickBot![/bold magenta]")
    console.print("   Ask about weather, request interesting facts, or just chat!")
    console.print("   Type 'exit' to quit.\n")

    while True:
        user_input = input("🗨️  You: ")
        if user_input.strip().lower() in {"exit", "quit"}:
            console.print("👋 [bold]Goodbye![/bold]")
            break

        response = process_input(agent, user_input)
        console.print(f"\n🤖 [bold green]LimerickBot:[/bold green]\n{response}\n")

if __name__ == "__main__":
    main()

Here’s a sample of the type of outputs you can get from this agent. To run it yourself, just install the required libraries and save the code into a agent.py file and run python agent.py in your terminal.

The agent always responds in a Limerick like format.

If I ask about the weather of some place, it automatically fetches it using the weather tool and still replies in a Limerick with the weather context.

I added another tool to my program to fetch random fun facts. When I ask for the same, the agent knows the correct tool to fetch random facts and calls it. Once it gets a fact, it uses the same to reply via a limerick.

If I ask about something unrelated to both weather data or random facts, it understands that it does not need to use any tools to reply and gives a direct limerick based on my input.

All of it running completely locally on my system.

Note that you have to use a base model that supports tool calling and structured outputs to get the best results.

If you are running the program on your own device, feel free to edit the system instructions, modify the tools, or add new tools to test different possibilities. Your imagination is the limit.

Here are some ideas:

1. Think of an agent that reads a stream of real-time tweets and creates responses to them based on the rich context of the user and even publishes them automatically.
2. How about an Agent that scrapes through the news and updates from the stock markets and the world of finance all day and sends you a summarized report of all important things, personalized to your context, directly to your WhatsApp?

Beyond personalized use cases, you can also have general-purpose task-completing agents, like a coding agent or a marketing agent that gets specific tasks done.

The latest state-of-the-art models are so good at getting the context or intent of the user that you can create many diverse agentic workflows.

Any structured process can effectively be transformed into an autonomous agentic workflow, given the right tools, APIs, and functions. I think this is a fundamental change in the way we work and get things done.

It will be very interesting to see what happens in the near future. Will it be an agent-first world or a human-in-the-loop using agents to gain more productivity?