How to Build a Real AI Agent (Not Just a Chatbot)

Explore how delivering the right information at the right time transforms your LLMs into really powerful AI agents.

Figure generated from the ChatGPT, showing the overview of context engineering.

The Rise of Context Engineering

In recent years, a new term has taken the AI world by storm: Context Engineering.

Championing this concept are AI thought leaders like Andrej Karpathy and Shopify CEO Tobias Lütke, who view it as a core competency in the age of intelligent agents. Its emergence stems from the limitations of traditional prompt engineering, which is often misunderstood as simply crafting clever instructions, while neglecting the underlying architecture and complexity of delivering the right information to LLMs.

Screenshot taken Andrej’s X, showing his support to context engineering.

With the rise of AI agents, and OpenAI President Greg Brockman proclaiming 2025 as “the year of AI agents”, delivering high-quality context has become the single most important determinant of whether an agent succeeds or fails.

Screenshot taken Greg’s X

What Makes Up “Context” for LLMs?

Context refers to everything the model sees before generating a response. That includes:

• System Prompt / Instructions: Initial rules and examples guiding your model behavior
• User Input: The user’s live task or question
• Short-Term Memory / Chat History: The current conversation so far
• Long-Term Memory: Persistent facts across sessions — user preferences, past summaries, or saved project data
• Knowledge Retrieval (RAG): Dynamic, real-time information retrieved from databases, APIs, or documents
• Tool Definitions: Functions available to the LLM, such as send_email() or check_inventory().Actually, thanks to the Claude, this is what MCP is doing, and this is the reason why MCP is so popular and well-know till now.
• Tool Responses: Outputs returned from those tools that become part of the evolving context
• Structured Output Definitions: Predefined response formats like JSON or Markdown
• Global State / Workflow Context: Shared memory or task progress data across a multi-step process or agent workflow

The combination from the above 9 elements forms the foundation of all intelligent applications built on LLMs today.

Sketch taken from the original article talking about context engineering. See reference at the end of this article.

Unlike static prompt engineering, which aims to craft the perfect sentence, context engineering is about building dynamic systems that deliver just the right knowledge, at just the right time, in just the right format — enabling the model to perform effectively without being overwhelmed or misled.

[embed]CK Huang Hello subscribers, I am CK, a passionate biologist and enthusiasm in AI world. Publishing AI, biology, and my…buttondown.com

A Real Example: Why Context Matters More Than Code

Here’s a practical example. Suppose you’re building an AI agent, which is used to reply to your emails, and now you receive the following email:

“Hey, just wondering if you’re free for a quick catch-up tomorrow?”

A basic AI agent — even with working codes — sees only this sentence and replies generically:

“Thanks for your message. I am available tomorrow. What time works for you?”

Now contrast that with a context-rich agent. Its code isn’t smarter, but it retrieves the following before generating a reply:

• Your calendar: Shows you’re fully booked tomorrow
• Past emails: Indicates whether to use formal or casual tone
• Contact list: Identifies the sender as a key partner or not
• Available tools: Like send_invite() or check_availability()

The final response is:

“Hey Jim! I’m packed tomorrow — back-to-back meetings all day. But I’m free this Thursday morning. I’ve sent over a calendar invite — let me know if that works!”

What made the second agent great wasn’t the language model’s IQ — it was the context engineering behind the scenes.

Key Strategies and Techniques in Context Engineering

To build high-performing AI agents, several technical strategies are becoming standard:

• Knowledge Base & Tool Awareness: The agent must know what tools and knowledge sources it can access before it can choose the right one.
• Context Compression & Ordering: LLMs have limited context windows. Techniques like summarization and recency-based ranking help maximize relevance while reducing length.
• Long-Term Memory Retrieval: Conversation history or facts saved across sessions is essential. Frameworks like LlamaIndex offer memory modules (e.g., VectorMemoryBlock, StaticMemoryBlock) for fine-grained control.
• Structured Information as Context: Feeding the model structured data (like tables or JSON) enhances clarity and efficiency. Tools like LlamaExtract help extract key fields from messy documents.
• Isolated Context in Multi-Agent Systems: Sub-agents can perform isolated tasks and return just the results, saving the lead agent’s context window and improving scalability.

Final Thoughts: Prompting Is Not Enough

For most people, prompt writing is a good starting point. As you move toward building real-world AI tools, prompt engineering becomes necessary. But if you’re designing interactive AI agents that rely on real-time tools, memory, or workflows — you’ve now entered the realm of context engineering.

This emerging field sits at the heart of LLM application design. It’s no longer about finding the “perfect prompt” or using the “most powerful model,” but about

delivering the right information, tool, and memory — in the right moment.

If we focus on building this infrastructure properly, artificial general intelligence (AGI) isn’t just a dream. It’s a roadmap.

If this article helped clarify the difference between prompts and context — and why it matters in building AI agents — hit that 👏 50 claps to help others discover it too.

Reference

Context Engineering — What it is, and techniques to consider