Video Summary — AI Agents Masterclass with Chi Wang, Founder of AutoGen

Introduction

I came across an interesting interview from October 2024 with Chi Wang, who is the founder of the AutoGen agentic framework. This post summarizes the key points from the video.

[embed]

Analysis of Transcript Excerpts — Key Points & Concepts:

Session Overview: The session is an “AI Agents Masterclass” with Chi Wang, the founder of AutoGen. It covers multi-agent frameworks, how AutoGen works, benefits/drawbacks of multi-agent vs. single-agent approaches, and includes “hot takes” on OpenAI’s new Swarm library. It’s part of a series, with more courses planned focusing on “nuts and bolts building” of agent systems.

AutoGen’s Core Concept:

AutoGen allows users to define agents and define how they can interact. Agents have their own prompts (instructions/roles) and can have tools attached. The goal is to provide a simple concept as an abstraction for developers to reason about complex applications, requiring learning only a few things. It encourages thinking about decomposing tasks like a human team3. AutoGen offers both high-level interfaces and lower-level controls (send/receive) for complex behaviors.

Positioning:

AutoGen is a higher-level framework compared to LlamaIndex workflows or LangGraph. It focuses on multi-agent definition and structuring interactions, sitting “a layer above” those that define single agents or specify exact workflows “in the weed’s”.

Multi-Agent Interaction Patterns:

Different organizational structures exist:

◦ Joint Chat: Agents talk freely.

◦ Hierarchical: One agent talks back and forth with another.

◦ Sequential Chat: Defined order like a pipeline, moving from one set of agents to the next4…. Analogous to defining steps for a team.

◦ Group Chat: Multiple agents converse, and a group chat manager decides who speaks at each turn, acting as a “super router”. This is considered the most autonomous orchestration method. Advanced features include transition constraints (hardcoded rules, natural language instructions) for balancing automation and domain expertise.

Agent Definition & Communication:

Agents are defined by their task, prompt, system message, model config, and optional tools. Communication is primarily text back and forth. Agents are composed by “starting a chat”.

Interoperability:

Interoperability between frameworks might be less problematic than usual because text is a common input/output unit. An agent or framework outputting/inputting text could communicate with another. While swapping an agent directly into another framework’s chat might be challenging, composing units via text interfaces is feasible. LlamaIndex agents can be used with AutoGen, for example, for RAG within an AutoGen group chat. Interoperability depends on whether external concepts can be “wrapped in an agent way”.

Termination:

An LLM agent can be instructed via its system message to say a specific termination word/phrase upon task completion. The other agent needs to recognize this (e.g., via predefined rules checking for the word, even with slight variations for robustness). This applies to multi-agent chats. We can also use max_turns to control this.

User Interaction:

Agents can ask users for clarification

◦ User Proxy Agent: An agent that doesn’t perform tasks but prompts the user for input and sends it back. It’s part of the chat, and the user is prompted when the group chat manager decides it’s the User Proxy’s turn based on defined conditions.

◦ Tool: A function that pauses, asks for human input, and returns the response. Can be added to any LLM agent, allowing the agent to proactively decide when to ask the user via a function call.

◦ Difference: Group chat manager decides turn (User Proxy) vs. agent decides via function call (Tool).

Case Study (Simple Assistant):

Building a multi-task assistant (docs, data analysis, SQL). Initial attempt using a single assistant with tools (mimicking a lower-level approach) worked but responses were “not as good”. A second attempt using the Group Chat multi-agent pattern with individual agents for each task led to better performance and responses due to more iteration among agents. This highlighted the need to shift mindset to multi-agent flows. The strength of multi-agent is improving results through iteration.

Observability & Debugging:

Tools like Arize Phoenix visualize multi-agent traces…. Traces show the chain of LLM calls back and forth. This is “really helpful” for debugging and understanding where issues occur. Debugging multi-agent is more complicated due to non-determinism and many parameters (“knobs”). Getting prompt wording right is crucial.

OpenAI Swarm Comparison (“Hot Take”):

Swarm is a recent multi-agent release

A notable aspect is its simple agent transition via function calls. AutoGen had a similar early example (math problem solver) using function calls to create nested chats. The similarity is using function calls for handoff. The difference is AutoGen’s nested chats can return, while Swarm’s transition seems sequential. AutoGen can implement a Swarm-like pattern using group chat and a state flow/FSM transition condition based on a user-defined function returning the next agent. Handoffs and flow control are key areas for development.

Challenges & Unsolved Problems:

◦ The biggest bottleneck is choosing the right patterns and composing them for optimal results (success rate, cost, latency, debuggability, maintenance). This is a difficult problem like choosing ML models/hyperparameters.

◦ Basic building blocks are close to complete/mature, shifting focus to organization and navigating the solution space.

◦ Potential drawback: multi-agent is harder to control.

◦ Debugging is more complicated.

◦ Termination can be an extra concern.

◦ Managing context and information flow is critical. Giving less capable agents minimal necessary context is generally better, as too much can be distracting even with long context windows. Hierarchical routing can help manage complexity. Combining different AutoGen patterns (group chat for shared context, sequential/nested for separated context) helps manage context appropriately. Deciding on separate RAG per agent vs. shared depends on information needs.

Evaluations:

Involves decomposing systems into tasks/components and building performance tests (code-based or LLM-as-a-judge). LLM-as-a-judge is common for text. Focus evals on critical skills or known issues.

Ready to Dive Deeper?

The multi-agent world is complex and fascinating. Frameworks like AutoGen are providing powerful tools to explore this space, but they also require a new way of thinking and introduce novel challenges in debugging and optimization. Understanding these concepts is crucial for anyone looking to build the next generation of AI applications.

For a more in-depth look at AutoGen, its patterns, and the future of multi-agent systems, watch the full masterclass session!