Using LangChain and Pydantic to Handle LLM Output More Reliably

Blog Thumbnail

Most people who’ve used a language model in a project, even briefly, have seen this problem. You ask the model for structured output — maybe JSON, maybe a simple itemized list — and instead, you get a mix of formatting, explanation, and sometimes an apology. The output might be close, but not parseable, or it might be wrapped in markdown, or start with something like “Sure! Here’s the answer:”

That works fine in a chat window. It doesn’t work when you’re building a system.

You can try to engineer better prompts and hope the model listens. Or you can approach this the same way we’ve approached all kinds of unpredictability in software: by setting structure, validating output, and enforcing contracts.

If you’re using LangChain already, this problem and its solution show up together. Here’s one way to deal with LLM output predictably — using Pydantic models, LangChain prompt tools, and optionally two models instead of one.

Why Pydantic?

If you’ve built Python APIs or worked with FastAPI, you’ve used Pydantic for defining schemas — typically input or response models. But it works just as well on output from a language model. You define the structure you want: types, fields, and shapes.

Let’s say you’re asking a model to answer a question and include some source links. Structured data might look like this:

{
    "question": "What is the capital of France?",
    "answer": "Paris",
    "sources": ["https://en.wikipedia.org/wiki/Paris"]
}

You can create a matching Pydantic model like this:

from pydantic import BaseModel
from typing import List

class Answer(BaseModel):
    question: str
    answer: str
    sources: List[str]

By itself, that doesn’t do much. But once this connects to a parser and some instructions, it lets you strictly control what the model returns — or verify if it didn’t return it correctly.

Building the Prompt

LangChain gives you a way to build prompt templates and include formatting guides. These formatting guides come from Pydantic via a parser LangChain provides.

Here’s how that gets set up:

from langchain.output_parsers import PydanticOutputParser
from langchain.prompts import PromptTemplate

parser = PydanticOutputParser(pydantic_object=Answer)

prompt = PromptTemplate(
    template=(
        "Answer the question below. Provide the output strictly as JSON, matching this format:\n"
        "{format_instructions}\n\n"
        "Question: {question}"
    ),
    input_variables=["question"],
    partial_variables={"format_instructions": parser.get_format_instructions()},
)

This reads like a human instruction — and models like GPT-4 tend to follow it pretty closely.

Calling the Model

Here you’re just executing a chain: feed the prompt to the model and apply the parser.

from langchain.chat_models import ChatOpenAI

llm = ChatOpenAI(model="gpt-4", temperature=0)

chain = prompt | llm | parser
result = chain.invoke({"question": "What is the capital of France?"})

If the model follows the instructions exactly (which you should check), result will be an instance of the Answer class, usable like any other Python object.

But Models Still Drift

Even with formatting guidelines, language models make mistakes. Sometimes they include an explanation before the JSON block. Sometimes formatting issues — like missing commas or bad quotes — cause parsing to fail.

You can handle these in a few ways. One is to catch the exception and retry. Another — better for cost — is to delegate repair work to a smaller model.

LangChain provides OutputFixingParser, which is a wrapper around your existing parser that uses another model like GPT-3.5 to fix or re-output response data this time correctly.

Here’s how that gets wired:

from langchain.output_parsers import OutputFixingParser

fallback_model = ChatOpenAI(model="gpt-3.5-turbo", temperature=0)
fixing_parser = OutputFixingParser.from_llm(parser=parser, llm=fallback_model)

safe_result = fixing_parser.parse(result.content)

Now you have a path where GPT-4 can do the heavy lifting — reasoning, answering, contextualizing — and GPT-3.5 can handle breakdowns related to formatting.

This split makes sense for cost reasons and often improves overall reliability.

Why This Pattern Works

Most developers building anything serious with LLMs will end up needing structured output — clean, consistent data the rest of their pipeline can use. Key-value pairs. Objects. Lists. JSON structures that don’t fall apart when passed to a frontend or written to a file.

Trying to force LLMs to behave through prompts alone works up to a point. But as complexity increases or scale becomes a factor, you need real structure.

Using Pydantic and LangChain together is an effective answer to that problem — and lets you lock in that structure without needing dozens of custom validators.

It also opens the door to deeper things — composing object-based responses, generating from custom schemas, tracing where breakage happens.

It’s not flashy. But it avoids a lot of brittle code and chaotic debugging.

And that usually pays for itself fast.

✨ Thanks for reading! I’d love to hear your thoughts — drop a comment below and let’s keep the conversation going.

Stay connected with me here:

• 🎯 Topmate: topmate.io/yash0307jain
• 🔗 LinkedIn: linkedin.com/in/yash0307jain
• 💻 GitHub: github.com/yash0307jain
• 🌟 AlgoMart (Follow for exciting projects!): github.com/AlgoMart

Until next time — let’s keep creating, sharing, and growing together. 👋