Your AI Agent Has Goldfish Brain. Here Are Two Ways to Fix It in Microsoft Agent Framework.

Your in-house counsel tells your AI contract review agent: “We never accept auto-renewal clauses, and we cap liability at 12 months of fees. Always.” Twenty minutes later they paste a new vendor MSA and ask the agent to flag the risky terms.

Your agent misses the auto-renewal on page 14 and waves through an uncapped liability clause. Welcome to the goldfish agent.

The Problem: Every Conversation Starts From Zero

Most AI agent tutorials hide the same dirty secret. The model only sees what you stuffed into the current prompt. When the session ends, the chat history dies with it. Your agent has no idea who the user is, what their playbook looks like, or what they told you yesterday.

That works fine for a “Hello World” chatbot. It falls apart the moment you build a contract reviewer, a legal research assistant, a tutor, or anything where the user expects the agent to remember them. A lawyer tells you on Monday that the client always rejects governing-law clauses tied to Delaware. By Tuesday afternoon they expect the agent to flag it on sight.

What you actually need is three things: persistence so client positions survive the session, relevance so you recall the right clause stance at the right time, and isolation so user A’s playbook never leaks into user B’s review. That’s a memory layer. And in Microsoft Agent Framework 1.0, you can wire one in two completely different ways.

Refer working code repository at github.com/imniteen/Microsoft-Agent-Framework-MyAgents.

Two Shapes of Memory

Same use case, two architectures. Same demo, very different ergonomics.

The src/mem0 folder uses Mem0 as an external memory service. The agent talks to it via function tools and decides when to read and when to write.

The src/azure-ai-foundry-memory folder uses Azure AI Foundry Memory as a context provider. The agent never thinks about memory at all. The provider handles it before and after every turn.

Both work. Which one fits depends on how much control you want and where your data lives.

Option 1: Mem0 as a Tool the Agent Calls

Mem0 is a hosted memory platform. You hand it text and a user_id, it handles embedding, deduplication, and semantic search. The cleanest way to plug it into Agent Framework 1.0 is to wrap the Mem0 client in a small class and expose three methods as function tools:

class MemoryTools:
    def __init__(self, client: MemoryClient, user_id: str):
        self.client = client
        self.user_id = user_id
        self._user_filter = {"AND": [{"user_id": user_id}]}

    def save_memory(self, information: Annotated[str, Field(...)]) -> str:
        """Save important user information to long-term memory."""
        self.client.add(
            [{"role": "user", "content": information}],
            user_id=self.user_id,
        )
        return f"Saved to memory: {information}"

    def search_memory(self, query: Annotated[str, Field(...)]) -> str:
        """Search long-term memory for relevant information about the user."""
        results = self.client.search(query, filters=self._user_filter)
        memories = _extract_memories(results)
        if not memories:
            return "No relevant memories found."
        return "\n".join(f"- {m['memory']}" for m in memories)

Three methods become three tools. Register them on the agent and you’re done:

agent = Agent(
    client=client,
    name="PersonalAssistant",
    instructions=AGENT_INSTRUCTIONS,
    tools=[
        memory_tools.save_memory,
        memory_tools.search_memory,
        memory_tools.get_all_memories,
    ],
)

Beautiful, right? Except the instructions block is doing a ton of hidden work:

1. At the START of every conversation turn, call 'search_memory' with a query
   relevant to the user's message to recall any useful context.
2. When the user shares standing positions, redlines, or playbook rules,
   call 'save_memory' to store them.
3. When the user asks "what do you know about my preferences?", call 'get_all_memories'.
4. Weave remembered playbook rules naturally into your reviews.

Without that prompt, GPT-4o will happily ignore your memory tools and re-derive every clause stance from scratch. The agent has to be told, every turn, to read and write. That cost is real: more tokens, more latency, more chances for the model to drift when the prompt gets long.

Option 2: Foundry Memory as a Transparent Context Provider

Foundry Memory takes the opposite approach. No tools. No workflow prompt. You attach a context provider that wraps the agent. Before each turn the provider pulls relevant memories and injects them. After each turn it extracts new ones and stores them. The model never sees a tool call.

Three pieces matter. First, create a memory store with a policy:

options = MemoryStoreDefaultOptions(
    chat_summary_enabled=False,
    user_profile_enabled=True,
    user_profile_details=(
        "Remember the user's standing client positions, contract clauses "
        "they routinely reject, preferred jurisdictions and governing law, "
        "liability caps, and risk tolerance. "
        "Avoid sensitive data such as client identifying information, "
        "billing records, or privileged matter notes."
    ),
)
memory_store = await project_client.beta.memory_stores.create(
    name=memory_store_name,
    description="Contract review playbook memory",
    definition=MemoryStoreDefaultDefinition(
        chat_model=os.environ["FOUNDRY_MODEL"],
        embedding_model=os.environ["AZURE_OPENAI_EMBEDDING_MODEL"],
        options=options,
    ),
)

Notice you tell Foundry what to remember and what to avoid. The extraction is policy-driven, not prompt-driven. The agent never gets a list of rules.

Second, wire the provider onto the agent:

memory_provider = FoundryMemoryProvider(
    project_client=project_client,
    memory_store_name=memory_store.name,
    scope=MEMORY_USER_SCOPE,
    update_delay=0,
)

async with Agent(
    name="PersonalAssistant",
    client=client,
    instructions="""You are a contract review assistant with long-term memory.
Memories from previous matters are automatically provided to you as context.
Use them to apply the user's playbook to every review.""",
    context_providers=[
        memory_provider,
        InMemoryHistoryProvider(load_messages=False),
    ],
) as agent:
    ...

No tools=[...]. The agent just runs. The instruction block is half the length of the Mem0 one, with zero workflow rules. The provider does the heavy lifting.

Third, you can inspect what got stored:

res = await project_client.beta.memory_stores.search_memories(
    name=memory_store.name,
    scope=MEMORY_USER_SCOPE,
)
for m in res.memories:
    print(m.memory_item.content)

Run the demo and entries like “The user rejects auto-renewal clauses by default” and “The user caps liability at 12 months of fees” materialise on their own. Nobody told the agent to save them. The provider extracted them.

The Side-by-Side

AspectMem0Foundry MemoryHow memory happensAgent calls toolsProvider auto-handlesPrompt complexityHigh, you write the workflowLow, “memories are in your context”StorageMem0 cloudAzure AI Foundry storeAuthMem0 API keyAzure AD (DefaultAzureCredential)EmbeddingsManaged by Mem0You deploy your ownVisibilityEvery read and write shows in the traceHidden inside the providerBest forFine-grained control, non-Azure stacksAzure shops that want managed memory

Trade-offs

Mem0 gives you transparency at the cost of prompt weight. Every turn pays for memory tool calls in tokens and latency, and the agent can drift if your instructions slip. Easier to swap out, harder to keep clean. For a contract reviewer where you need to defend in court why the agent flagged or missed a clause, that trace is gold.

Foundry Memory gives you a managed service at the cost of visibility. You don’t see when memory was read or written. Memory stores are still a beta feature without a portal page, so today you inspect them through the SDK only. If a senior partner asks why the agent applied last quarter’s playbook rule to today’s review, that’s harder to answer.

There’s no winner. Pick the trade-off that matches your team and your regulator.

Key Takeaways

1. Memory is not a feature you bolt on at the end. It changes how you write the agent.
2. Tool-based memory (Mem0) gives you control and visibility. Every read and write sits in the trace, which matters when you have to defend the agent’s decisions to a partner or a regulator.
3. Provider-based memory (Foundry) gives you a managed service. The framework absorbs the policy, but you lose easy inspection.
4. The instruction block is the tell. If your memory prompt has a numbered workflow, you’re using tools. If it just says “memories are in your context”, you’re using a provider.
5. Microsoft Agent Framework 1.0 supports both shapes cleanly. Pick based on your stack, not on what the framework forces.

Try It Yourself

git clone https://github.com/imniteen/Microsoft-Agent-Framework-MyAgents.git
cd Microsoft-Agent-Framework-MyAgents

# Mem0 path
cd src/mem0
python -m venv .venv && .venv\Scripts\activate
pip install -r requirements.txt
python main.py

# Foundry Memory path
cd ..\azure-ai-foundry-memory
python -m venv .venv && .venv\Scripts\activate
pip install -r requirements.txt
az login
python main.py

Tell the assistant your client never accepts auto-renewal clauses and caps liability at 12 months of fees. Then paste a vendor agreement that includes both and ask whether it’s safe to sign. The first reply tells you whether your agent has a memory or a goldfish brain.

Code, issues, and PRs welcome at github.com/imniteen/Microsoft-Agent-Framework-MyAgents.

An agent that forgets your client’s redlines is a junior associate who skipped the playbook. Build the memory in.

AgenticAI #MicrosoftAgentFramework #Mem0 #AzureAIFoundry #AIEngineering