Part 9 — From Basic Agent to Structured Extraction in Microsoft Foundry

In this blog, we build the foundation step by step using Microsoft Foundry — transforming a simple prompt-based agent into a structured invoice processing component ready for enterprise workflows.

Why Start with Foundations?

Invoice processing may look simple at first glance:

• Extract vendor
• Extract amount
• Validate total
• Approve or reject

But in reality, invoices contain:

• Mixed layouts
• Different currencies
• Variable table structures
• Edge cases
• Policy requirements

Jumping directly into multi-agent orchestration without mastering these fundamentals leads to fragile systems. So we begin properly.

Microsoft Foundry enables this progression by combining:

• Model hosting
• Tool orchestration
• Thread-based memory
• Multi-modal intelligence
• Enterprise governance

It allows us to move from simple prompting to production-grade agent systems.

GitHub Repo, Sample Data & steps to produce demo :

alpaBuddhabhatti/invoice-agent-demo (API key + Agent Framework (Azure OpenAI endpoint style))

alpaBuddhabhatti/invoice-agent-demo-Keyless (Keyless (Entra ID) + Microsoft Foundry Project endpoint + Managed Agents SDK )

Stage 1 — Creating a Minimal Viable Agent (MVA)

We start with the simplest possible invoice agent.

agent = Agent(
 client=get_chat_client(),
 instructions="Summarize invoice data."
 )

 result = await agent.run(
 "Invoice INV-1001 from Contoso for 1200 USD"
 )

What This Gives Us

• Natural language understanding
• Flexible interpretation of invoice text
• Fast experimentation

What It Lacks

• Structured outputs
• Validation
• Integration capability
• Workflow automation

At this stage, it is a smart assistant — not a system component. But it proves the model understands invoice semantics.

File — step1_basic_agent.py

Stage 2 — Adding Conversational Memory with Threads

Real workflows are rarely one-off prompts.

Users ask follow-up questions:

• What is the total amount?
• Who is the vendor?
• When is the due date?
• Is it approved?

Microsoft Foundry enables thread-based memory:

File — step2_thread_memory.py

Microsoft Foundry allows thread-based memory:

thread = agent.get_new_thread()

 await agent.run(
 "Invoice from Contoso for 1200 USD",
 thread=thread
 )

 result = await agent.run(
 "What is the total amount?",
 thread=thread
 )

Now the agent maintains context across interactions.

🧠 What Memory Means in Our Invoice Agent

Memory is not just conversation history. It is structured context preserved during invoice processing. For our invoice agent, memory maintains:

• Extracted invoice fields (vendor, total, date, tax)
• Validation results
• Business rule decisions
• Processing state across tool calls

Without memory, each step would require the full invoice text to be re-sent to the model. With memory, the agent works incrementally and intelligently.

In our implementation:

• Thread = Execution context for one invoice
• Memory = Structured data extracted during processing

Each invoice runs in its own thread.Memory keeps extracted facts isolated within that thread — preventing cross-invoice contamination.

This Enables

• Multi-turn conversations
• Context-aware responses
• Interactive financial workflows

However, responses are still unstructured text — and unstructured text cannot be reliably stored in databases or ERP systems. So we move forward.

Stage 3 — Introducing Tools for Structured Extraction

This is where the transformation begins. This is the turning point. Instead of only generating responses, the agent can call Python functions (tools).

@tool(name="extract_invoice")
 def extract_invoice(text: str) -> dict:
 return {
 "vendor": "Contoso",
 "amount": 1200,
 "currency": "USD"
 }

Now the agent workflow becomes:

1. Read invoice text
2. Call extract_invoice()
3. Return structured data

File — step3_invoice_tool.py

🛠 Where Tools Fit in Our Invoice Workflow

Our processing flow now becomes:

• Extract invoice data
• Validate business rules
• Return structured JSON
• Prepare for enterprise scaling

Tools support each step. Instead of asking the LLM to “think about validation,” we define deterministic rules such as:

• Subtotal + Tax = Total
• Vendor exists in approved list
• Total < approval threshold

The validation tool enforces logic programmatically. In financial systems, hallucination is unacceptable. Tools provide control, determinism, and reliability.

We have now moved from:

Chatbot → Workflow component

This is the first major architectural shift.

Why Structured Output Matters

Structured output enables:

• Database storage
• ERP integration
• Reporting pipelines
• Power BI dashboards
• Audit logging

Without structure, automation breaks. With structure, systems scale.

Stage 4 — Adding Business Rule Validation

Extraction alone is not enough. Enterprise systems require policy enforcement.

Example rule:

• Invoices ≤ $10,000 → Auto-approve
• Invoices > $10,000 → Require approval

def validate_invoice(amount: int, currency: str) -> str:
 if amount > 10000:
 return "REQUIRES_APPROVAL"
 return "APPROVED"

Now the workflow becomes:

1. Extract structured data
2. Validate business rules
3. Assign approval status

This introduces:

• Decision automation
• Policy enforcement
• Compliance alignment

At this stage, we are no longer “chatting with invoices.” We are building a financial workflow engine.

Architectural View of the Foundation

By Stage 4, your system contains:

• 🧠 AI Agent (semantic understanding)
• 🧵 Thread memory (context)
• 🛠 Tool layer (structured extraction)
• 📐 Validation logic (business policies)

This layered design creates a clean separation:

This structure is critical. Because in Part 10, we will extend it into a multi-agent enterprise system.

Why This Foundation Matters

Many AI demos stop at “Look, it extracts data.”

Enterprise systems require:

• Deterministic outputs
• Repeatable validation
• Structured storage
• Traceability
• Governance readiness

Without this foundational design:

• Multi-agent systems become chaotic
• Scaling becomes risky
• Compliance becomes difficult

Strong foundations prevent fragile architectures.

What’s Next (Part 10)

In Part 10, we will evolve this foundation into:

• Vision-based invoice understanding
• Specialized agent roles
• Multi-agent orchestration
• Enterprise-ready architecture
• Production governance considerations

We will move from:

Single Agent → Multi-Agent System Workflow Component → Enterprise AI Architecture

Final Thought

AI transformation does not start with complexity. It starts with clarity.

By designing:

• Clear responsibilities
• Structured outputs
• Deterministic validation

You create systems that scale safely. And that is how intelligent invoice processing begins — not with hype, but with architecture.