One Agent, Every Channel: Microsoft Foundry and Twilio, End to End

A technical walkthrough of hosted agents, the Agent Connect SDK, and the architecture that gives a voice call and a text message one shared memory.

One Agent, Every Channel: Microsoft Foundry and Twilio, End to End

This article walks through how to build an AI agent that works across voice and text without losing track of who it is talking to. I break down the architecture that connects Microsoft Foundry, which supplies the models and the agent runtime, with Twilio, which supplies the phone, SMS, and chat channels, using an open source SDK called Twilio Agent Connect. By the end, you will understand each component, how they fit together, why the runtime uses a new kind of compute built for agents, and how to deploy the whole thing yourself with a single command.

Reference here.

The problem worth solving

Most conversational agents have amnesia. You tell one over text that your flight is delayed, then you call the same company an hour later and the voice bot greets you like a stranger. That gap is not a model problem. Modern models are strong and easy to reach. The gap lives in the layers around the model: the channels, the memory that should follow a customer across those channels, and the runtime that has to keep all of it reliable under real traffic.

The industry numbers make the stakes concrete. Gartner puts global AI spend on a path toward $2.5 trillion, up 44 percent year over year. At the same time, PwC data shows more than half of leaders reporting little to no return on that spend, and Forrester found only 6 percent of brands improved their customer experience scores in 2025. Spending is compounding while customer experience barely moves. The likely reason is that teams keep shipping a great model wired into a fragile experience, and the fragile experience is what customers feel.

Three fractures that break omnichannel agents

Three specific failure modes explain most of that fragility, and each one maps onto a part of the architecture below.

The first is memory loss. Traditional customer engagement forgets across channels. Last week’s email, this morning’s call, and last month’s website visit sit in separate systems that never compare notes. To the customer it feels like talking to a company with no short term memory.

The second is a disjointed experience. Voice and digital are usually built by different teams on different stacks, so context does not travel between them. This is the delayed flight scenario: the company already texted you the delay, then the agent opens with “how can I help.” Every handoff starts from zero, and every restart chips away at trust.

The third is the infrastructure hurdle, and engineers tend to underestimate it. Standing up one clever agent in a notebook is easy. Running an agent that is omnichannel, keeps memory, holds the right knowledge, and stays reliable at scale is a real systems problem. The hard part is no longer writing the agent. The hard part is operating it.

These are not three separate problems. They are one problem, context, seen from the memory layer, the channel layer, and the runtime layer.

Two complementary stacks, one agent

The core design idea is that Foundry and Twilio are complementary rather than overlapping. Draw a line down the middle and the division of labor is clean.

On the Twilio side sits the omnichannel layer. Twilio owns the conversation: native low latency speech transport through Voice and ConversationRelay, plus enterprise grade global delivery across SMS, WhatsApp, RCS, and chat. Two Twilio products carry the context. Conversation Memory holds cross channel customer context, and Conversation Configuration routes every channel into one thread tied to a customer profile. That profile resolution is the quiet hero of the demo later on.

On the Foundry side sits the intelligence layer. Foundry owns the reasoning: the Voice Live API for unified low-latency speech, a broad model catalog spanning Azure OpenAI, Anthropic, Meta, and more, the Microsoft Agent Framework as an open SDK for building the agent your way, and an enterprise foundation with Microsoft Entra identity and platform governance. Foundry is where you build, optimize, and govern the agent.

Between them sits Twilio Agent Connect, the bridge. One open source SDK, two connectors, one to Twilio and one to Foundry, producing a single agent that behaves consistently on every channel. The goal is to move from concept to production with far less custom code, and it works because neither vendor asks you to rebuild the other’s half.

Deployment, take one: Container Apps

The first version of this integration shipped on Azure Container Apps, and it remains a supported option, so it is worth understanding before we get to what came next.

The topology is simple. Your Twilio channels connect over WebSockets into a container app on production-grade Azure infrastructure. The app persists session state in Azure Cosmos DB and calls your model in Foundry for inference. You get full control over the runtime, pluggable session persistence with file storage by default or Cosmos DB for horizontal scale, and the familiar operational surface of a container platform.

It works. But building it surfaced a lesson that reshapes the rest of the design: traditional compute was not built for agents.

Why traditional compute is the wrong shape for agents

This is the most important architectural point in the whole design, so it deserves room.

Container platforms and serverless functions were built for stateless web services and APIs, where many users safely share one instance because no request carries durable state into the next. Agents break every one of those assumptions. An agent session is long-lived and stateful per user. It writes files, it executes code, and it holds credentials while it works.

Now put two customers on the same shared instance. Customer A and Customer B are both talking to the agent, who is writing files and running tools on shared compute. You have just created an isolation problem: a prompt injection or a misbehaving tool in Customer A’s session becomes a blast radius that can reach Customer B’s data. On a stateless web compute, you patch around this by adding external databases for state, hand-rolling isolation that the runtime never gave you, and propagating identity yourself. You end up building around the platform instead of with it.

That is why Microsoft built a new compute primitive specifically for this pattern rather than stacking more scaffolding onto container compute. The distinction is the line between a demo that runs and a system you can operate.

Deployment, take two: hosted agents

Hosted agents in Foundry Agent Service are that new primitive. They give an agent session six properties that container compute cannot, and each property answers a specific failure above.

• Isolation. Every session gets its own hypervisor-isolated sandbox. Not process isolation, not a code execution only sandbox, but per-session VM-level isolation at cloud scale. This is the answer to the Customer A and Customer B problem.
• Cold starts. Sandboxes spin up in seconds with low, predictable variance, instead of the seconds to minutes swing of scaling containers from zero.
• Idle cost. Agents scale to zero and cost nothing while idle, then resume with the working directory intact. That suits messaging, where a reply might land hours later.
• State persistence. State is built in. The $HOME directory and the /files endpoint survive idle, so you are not provisioning a database just to remember the last two turns.
• Identity. Every deployed agent receives its own Microsoft Entra Agent ID at deploy time, plus on behalf of flows, instead of a shared service account. Identity becomes a per-agent property.
• Observability. End-to-end OpenTelemetry tracing, evaluations, and policies are built in, so you get traces, monitoring, and evals without wiring any of it yourself.

Because hosted agents reached public preview with WebSocket support and an Invocations API, Twilio could move its deployment target from container apps onto hosted agents. The fit is easy to describe in runtime terms: fast startup serves latency sensitive voice, and zero idle cost suits messaging that can wait. Two runtime traits matched to two channel types.

The hosted agents architecture, step by step

The new topology is far simpler than the container version, and every hop earns its place.

Twilio handles voice, SMS, and chat at the edge. Channel events flow over HTTP into an Azure API Management gateway. The gateway does three jobs before anything reaches your agent. It validates the Twilio signature so you know the request is real. It adds authentication. And it maps each Twilio conversation ID to a hosted agent session ID, which is the step that gives every conversation its own isolated sandbox. From the gateway, the request routes into that sandbox where your agent logic runs, and the sandbox calls Microsoft Foundry for model inference. The result is fully serverless: no compute to manage, no scaling to tune, no idle cost.

One detail is easy to miss and important to get right. There are two transport paths here. Omnichannel events such as SMS, chat, and control messages travel over HTTP. Voice rides a separate native WebSocket path, and that separation is the reason voice stays fast enough to feel like a real conversation. When a voice reply comes back in under a second, that WebSocket path is why.

There is a public and private URL nuance too. The API Management gateway URL is public, and it is the endpoint you paste into Twilio as a webhook. The hosted agent’s own URL is not public. You always point Twilio at the gateway, never at the agent directly.

Inside Twilio Agent Connect, the SDK

So what sits between your Twilio channels and your hosted agent? Twilio Agent Connect, or TAC, is a connector. The Microsoft package ships two connectors plus a set of primitives that you wire in with a system prompt, and you have a multi channel agent. Here is what each piece does.

The AgentFrameworkConnector routes Twilio events across voice and messaging directly into the Microsoft Agent Framework. It wires both channels into a single Agent Framework agent and exposes the hooks production demands: channel aware system prompts so voice and SMS can behave differently, session lifecycle management, and error handling. It also manages session persistence, with a CosmosDBAgentSessionStore for horizontally scaled production and a pluggable AgentSessionStore protocol so you can back sessions with Redis, Postgres, or anything else.

The VoiceLiveConnector provides direct integration with Foundry Voice Live. It streams Twilio audio into Voice Live’s unified speech and reasoning interface: one config object, sub second latency, and no separate speech to text or text to speech pipeline for you to build and maintain. This is where the voice path gets its speed.

Memory and context injection is the mechanism behind cross channel recall. An auto_retrieve_memory step pulls Twilio's Conversation Memory, and format_memory_context injects it into the user message before the model sees it. The default is tuned per channel: off for voice, where every millisecond of latency is audible, and on for SMS, where a beat of retrieval is invisible. That per channel default is a small choice with an outsized effect on how the agent feels.

Tools round it out. You can pass any Python function to the agent as a tool. A create_knowledge_tool helper wires in a Twilio knowledge base, and a create_memory_recall_tool exposes memory as an explicit tool the agent can choose to call, rather than only as background context.

The class that makes hosted deployment trivial is TACHostedAgentsApp. It connects the hosted agent request originating from Twilio into TAC, and it installs into your own Python code through pip. That class is what lets the same agent code run inside Foundry's serverless sandbox with almost no ceremony.

The demo, and why it matters

The demo is a simple airline scenario, and its simplicity is the point: it isolates the one capability everything above exists to deliver.

First, a voice call. The caller reaches the agent about flight 543 to Seattle, departing tomorrow. They ask to change to a window seat. The agent confirms they are currently in an aisle seat, offers the window seat, and completes the change on request. Twilio handles speech in and out, TAC bridges into the Foundry Agent Framework, and Foundry powers the reasoning. There is no perceptible delay, which is the WebSocket voice path doing its job.

Then the reveal. The same person sends an SMS from the same number: they forgot to add a bag. The agent replies that it sees them on flight 543, in seat 23A. Read that again. Seat 23A was chosen seconds earlier on the voice call. Different channel, different transport, different default for memory injection, and the agent still knows exactly who this is and what just happened. It quotes the bag fee, confirms, and finishes.

That continuity is profile resolution at work. Because both interactions share a phone number, Twilio resolves them to one customer profile, Conversation Memory carries the seat change forward, and context injection hands it to the model on the SMS turn. The three fractures from the start, memory loss, disjointed experience, and infrastructure, are each closed by a specific component, and the demo makes all three closures visible in about ninety seconds.

A single azd up command provisions the API Management instance and a hosted agent with TAC running directly inside it. It takes five to ten minutes, and the deployed agent inherits the full hosted agents surface out of the box: traces, monitoring, evaluations, and a playground, none of which you had to build. The demo runs on a compact GPT model, which reinforces the point that this is an architecture win, not a raw model horsepower win.

Where ConversationRelay fits in

One clarification prevents a common misconception. The native integration does not replace Twilio ConversationRelay. Under the hood, the voice path still uses ConversationRelay, now embedded inside the Twilio Agent Connect repository. Speech-to-text and text-to-speech in the Voice Live connector are still handled by ConversationRelay.

So what changed? ConversationRelay was the original approach because the integrated deployment options did not exist yet. Once Microsoft shipped WebSocket support and Twilio signature validation on hosted agents, a direct integration became possible, and it makes voice calls faster while letting Microsoft own the WebSocket infrastructure for teams that would rather not. If you already built on ConversationRelay directly and you want to own that infrastructure, that path is still fully valid. The native integration is an option that adds evaluations, observability, and a managed deployment, at no extra charge beyond the channels and Foundry services you already pay for.

Extending it, and the multi-tenant question

For customization, the repository ships generic examples (“you are a helpful agent”) with README recipes for the things you will most likely change: swap the personality prompt, connect a knowledge source, add a tool, change the model, attach data, add evaluations. You can do this in the TAC repository or inside Foundry’s hosted agent building experience, which gives you two on ramps depending on whether you prefer code or portal.

The one open design question is multi-tenant provisioning for independent software vendors. With the pre-integration version of Twilio Agent Connect, the guidance was a separate server, and likely a separate subaccount, per company. How that maps best onto hosted agents, one project and resource per tenant versus a shared project with tenant scoping, is still settling. If you are building a true multi-tenant product on this, that is the decision to pin down early.

Final Thoughts

The interesting shift here is where the difficulty lives. A few years ago, the hard part of a conversational agent was the model. Today, the models are strong and easy to reach, so the real work moves to everything around them: keeping memory consistent as a customer jumps from a call to a text, isolating one user’s session from another, and running all of it without babysitting servers.

This integration lines up a clear answer for each of those. Foundry’s hosted agents give every session its own sandbox and handle scaling, identity, and tracing for you. Twilio Agent Connect carries the channels and the memory, so context follows the customer instead of getting stranded in one system. You write the agent once, and it behaves the same on voice and on SMS.

Resources

Microsoft Foundry Agent Service overview: https://learn.microsoft.com/en-us/azure/foundry/agents/overview?WT.mc_id=AZ-MVP-5000671

Hosted agents in Foundry Agent Service (concept): https://learn.microsoft.com/en-us/azure/foundry/agents/concepts/hosted-agents?WT.mc_id=AZ-MVP-5000671

Quickstart: deploy your first hosted agent: https://learn.microsoft.com/en-us/azure/foundry/agents/quickstarts/quickstart-hosted-agent?WT.mc_id=AZ-MVP-5000671

Migrate hosted agents to the refreshed public preview: https://learn.microsoft.com/en-us/azure/foundry/agents/how-to/migrate-hosted-agent-preview?WT.mc_id=AZ-MVP-5000671

Microsoft Agent Framework documentation: https://learn.microsoft.com/en-us/agent-framework/?WT.mc_id=AZ-MVP-5000671

Microsoft Agent Framework overview: https://learn.microsoft.com/en-us/agent-framework/overview/?WT.mc_id=AZ-MVP-5000671

Azure AI Voice Live API overview: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/voice-live?WT.mc_id=AZ-MVP-5000671

Build a voice agent with Voice Live and Foundry Agent Service: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/how-to-voice-agent-integration?WT.mc_id=AZ-MVP-5000671

What is Microsoft Entra Agent ID: https://learn.microsoft.com/en-us/entra/agent-id/what-is-microsoft-entra-agent-id?WT.mc_id=AZ-MVP-5000671

Azure Container Apps overview: https://learn.microsoft.com/en-us/azure/container-apps/overview?WT.mc_id=AZ-MVP-5000671

Azure API Management key concepts: https://learn.microsoft.com/en-us/azure/api-management/api-management-key-concepts?WT.mc_id=AZ-MVP-5000671

Azure Cosmos DB for NoSQL: https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/?WT.mc_id=AZ-MVP-5000671

Azure Developer CLI (azd) overview: https://learn.microsoft.com/en-us/azure/developer/azure-developer-cli/overview?WT.mc_id=AZ-MVP-5000671

Twilio Agent Connect for Microsoft (GitHub repository): https://github.com/twilio/twilio-agent-connect-microsoft

Microsoft Agent Framework (GitHub repository): https://github.com/microsoft/agent-framework

Twilio Agent Connect documentation: https://www.twilio.com/docs/conversations/agent-connect

Twilio: connect Microsoft Foundry agents to Twilio channels: https://www.twilio.com/docs/conversations/agent-connect/integrations/microsoft-foundry

Twilio tutorial: build a voice and SMS AI agent with Twilio Agent Connect and Azure: https://www.twilio.com/en-us/blog/developers/tutorials/integrations/voice-sms-ai-twilio-agent-connect-microsoft-azure

*-Dave R.*