What to know before using LLM frameworks — Part 3

Plan, then work

Last time we go through the basic of how LLM uses the tool. But it only use one tool per time. When things getting complex, it might not be able to handle it.

For example, a coding agent needs to use file read tool to read the design document before adding new feature to the file.

Sometimes a LLM cannot solve a complex problem with a simple prompt.

There are a lot of prompt technique to solve complex problem efficiently. In this post, I am going to talk about some common prompt engineering techniques, from simple to hard:

• Few-Shot prompting
• Chain-of-Thought prompting
• ReAct

Zero-Shot v.s. Few-Shot prompting

The first prompt technique is called “Zero-Shot” prompting. Zero-Shot prompting means we interact with the LLM without any examples of how to solve it. In the other hand, Few-Shot prompting means you will give few examples to the model.

For Zero-Shot example, you can ask it to rewrite a sentence with specific style, but a custom style that never appears on the public training dataset

User:
Rewrite this sentence in Yi style
"I can’t go to the party because I’m busy."

LLM:
In Yi, a dialect spoken in China, the sentence would be rewritten as:

"Jīn tā bù dēng guóyè."

The LLM will use what it learned from the training to predict the result and may not meet your expectation.

To make sure it understand what “Yi style” is, we can give few examples to the LLM and ask it again:

User:
Here are some example of rewriting sentence in Yi style:
"It’s very cold outside." -> "The air doth bite with bitter frost."
"I’m happy to see you." -> "My heart leaps with joy at thy presence."

Rewrite this sentence in Yi style
"I can’t go to the party because I’m busy."

LLM:
Here's a rewritten version of the sentence in Yi style:

"Though duty calls, my feet remain behind."

Now LLM can convert sentence to “Yi style” correctly(can you guess what the real style is? 😃)

This is called Few-Shot prompting, which gives the LLM few examples before asking it to perform a task. For this example, it is also known as 2-shot prompting since we gave 2 examples to it.

The Prompt Engineering Guide and the paper Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? provides few tips about when few-shot prompting works well:

• You can show the label space: Just seeing the types of outputs (e.g., “positive,” “neutral,” “negative”) helps the model orient itself.
• You provide examples from the same input distribution: Inputs that resemble the test data (in style, domain, or structure) boost performance.
• You maintain the format of input-label pairs: Even if the labels are random, the structure of examples helps the model infer the task.

Additionally, there are some cases that few-shot prompt might not work:

• Out-of-distribution inputs: If your examples come from a domain the model hasn’t seen (e.g., highly technical jargon or synthetic data), performance drops.
• No clear label space or format: If the prompt lacks structure or mixes formats, the model may not infer the task correctly.
• Tasks requiring reasoning or multi-step logic: For complex tasks like math or reasoning, incorrect or noisy examples degrade performance more sharply.

Chain-of-thought prompting

Unlike few-shot prompting, Chain-of-thought prompting(CoT) asks LLM work on a task step-by-step. Some research like Chain-of-Thought Prompting Elicits Reasoning in Large Language Models uses Few-Shot Chain-of-Thought, which provides some examples that contains steps to solve the problem.

1-shot v.s., 1-shot-CoT - Wei et al.

In the CoT paper mentioned above, there are few properties from CoT prompting:

1. Decomposes complex problem or task to small steps.
2. It is easier to user to understand how a model figureout the final answer.
3. CoT can be used for tasks like solving math problems, commonsense reasoning, and symbolic manipulation, and potentially be able to solve any tasks that humans can solve via language.
4. CoT prompting works well in few-shot settings by including examples with intermediate reasoning steps.

Another paper Large Language Models are Zero-Shot Reasoners shows that we can also perform CoT prompting with no example(Zero-shot-CoT).

The Zero-shot-CoT can be as simple as adding “Let’s think step by step”. The LLM will return steps before it provides the final answer.

Moreover, this paper suggests that we will break this into two prompts:

1. Reasoning: ask the LLM to generate steps
2. Answering: ask the LLM to give us the final answer based on steps.

For eample, we can send the first prompt like:

Q: On average Joe throws 25 punches per minute. 
A flight lasts 5 rounds of 3 minutes. 
How many punches did he throw?
A: Let's think step by step.

LLM will generate something like this:

In one minute, Joe throes 25 punds
In three minute, Joe throws 3*25 = 75 puches
In five rounds, Joe throws 5 * 75 = 375 punches

Now we can combine the question and the reason part and add additional prompt to ask the final answer:

Q: On average Joe throws 25 punches per minute. 
A flight lasts 5 rounds of 3 minutes. 
How many punches did he throw?
A: Let's think step by step.
In one minute, Joe throes 25 punds
In three minute, Joe throws 3*25 = 75 puches
In five rounds, Joe throws 5 * 75 = 375 punches

Therefore, the answer(arabic numerals) is

Then LLM will return “375” as the final answer.

In the same paper, they compares multiple prompting technique(zero-shot, few-shot, zero-shot-CoT, and few-shot-CoT) across different models and types of problem. Turns out the Zero-shot-CoT improves the performance significantly(compares to zero-shot and few-shot prompting).

Source: Large Language Models are Zero-Shot Reasoners

Although Few-Shot-CoT performs better than Zero-Shot-CoT, it requires more effort for engineer to craft examples for LLM, and you need to include different types of examples for different type of task.

Few-shot-CoT requires careful human engineering of a few prompt examples with specific answer formats per task, while Zero-shot-CoT requires less engineering but requires prompting LLMs twice.

Modern models this days, especially models with reasoning capability, works well with Zero-shot-CoT since it is already fine-tuned with many examples for different types of tasks. So engineer can focus on providing clear context and instructions instead of gathering possible eaxmples for LLM for certain tasks.

Reasoning + Acting

Thinking, and decide which tool to use!

Finally, we can use techniques below with tools to build applications like AI agent.

By using advance prompting like CoT, LLM can use one or more tools accurately. Especially tasks requires multiple steps and tools.

For example, we need to search shopping records from a user with user’s name, and we have following tools:

1. get_user_id(firstname, lastname)
2. get_shipping_records(user_id)

With 0-shot CoT, we can give following prompt(from: LangSmith Hub):

Answer the following questions as best you can. You have access to the following tools:

get_user_id(firstname, lastname)
get_shipping_records(user_id)

Use the following format:

Question: the input question you must answer
Thought: you should always think about what to do
Action: the action to take, should be one of [get_user_id, get_shipping_records]
Action Input: the input to the action
Observation: the result of the action
... (this Thought/Action/Action Input/Observation can repeat N times)
Thought: I now know the final answer
Final Answer: the final answer to the original input question

Begin!

Question: "Show me shopping records from Yi Tseng"
Thought:

Here is an example output from the model

I need to retrieve the user ID for Yi Tseng first to access the shipping records.  
Action: get_user_id  
Action Input: "Yi", "Tseng"  
Observation:

In this case, we can use a regular expression parser to parse the action result and send to the tool and fill the observation.

Once we have the result from the tool(12345678 in this example), we can send the updated prompt back to the model and get next action:

Answer the following questions as best you can. You have access to the following tools:

get_user_id(firstname, lastname)
get_shipping_records(user_id)

Use the following format:

Question: the input question you must answer
Thought: you should always think about what to do
Action: the action to take, should be one of [get_user_id, get_shipping_records]
Action Input: the input to the action
Observation: the result of the action
... (this Thought/Action/Action Input/Observation can repeat N times)
Thought: I now know the final answer
Final Answer: the final answer to the original input question

Begin!

Question: "Show me shopping records from Yi Tseng"
Thought: I need to retrieve the user ID for Yi Tseng first to access the shipping records.  
Action: get_user_id  
Action Input: "Yi", "Tseng"  
Observation: "12345678"
Thought:

Now we get following from the model:

I have retrieved the user ID for Yi Tseng. Now I can access the shipping records using this user ID.  
Action: get_shipping_records  
Action Input: "12345678"  
Observation: "[{'order_id': 'ORD123', 'date': '2023-02-15', 'items': ['item1', 'item2']}, {'order_id': 'ORD124', 'date': '2023-03-18', 'items': ['item3']}]"
Thought: I now have the shipping records for Yi Tseng.
Final Answer: The shipping records for Yi Tseng are: 1) Order ID: ORD123, Date: 2023-02-15, Items: item1, item2; 2) Order ID: ORD124, Date: 2023-03-18, Items: item3.

Note that the model also outputs the result from the tool because models can hallucinate the result.

In this case we can simply ignore all text after “Observation” and get the correct result from the tool.

Now we can send following back to the model and try to get final result:

Answer the following questions as best you can. You have access to the following tools:

get_user_id(firstname, lastname)
get_shipping_records(user_id)

Use the following format:

Question: the input question you must answer
Thought: you should always think about what to do
Action: the action to take, should be one of [get_user_id, get_shipping_records]
Action Input: the input to the action
Observation: the result of the action
... (this Thought/Action/Action Input/Observation can repeat N times)
Thought: I now know the final answer
Final Answer: the final answer to the original input question

Begin!

Question: "Show me shopping records from Yi Tseng"
Thought: I need to retrieve the user ID for Yi Tseng first to access the shipping records.  
Action: get_user_id  
Action Input: "Yi", "Tseng"  
Observation: "12345678"
Thought:I have retrieved the user ID for Yi Tseng. Now I can access the shipping records using this user ID.  
Action: get_shipping_records  
Action Input: "12345678"  
Observation: ['apple', 'orange', 'cake']
Thought:

And finally, the model can return the final answer like this:

I have successfully retrieved the shipping records for Yi Tseng, which include the items: apple, orange, and cake.  
Final Answer: The shopping records from Yi Tseng are: apple, orange, and cake.

Now we have a final answer, we can use our program to parse the result and output it with certain format to the user.

Let’s move to modern LLMs!

If you check the link from LangSmith hub for the ReAct prompt

In part 2, we talked about how modern LLMs work with tools and how easy we can use tools without creating parsers.

Modern LLMs also trained with ReAct way to solve tasks. It just use a different format.

For example, this is what it might looks like when using gpt-oss model:

<|start|>system<|message|>You are ChatGPT, a large language model trained by OpenAI.
Knowledge cutoff: 2024-06
Current date: 2025-08-19

Reasoning: medium

<|start|>developer<|message|># Tools

## functions

namespace functions {
// gets user id with firstname and lastname
type get_user_id = (_: {
firstname: string,
lastname: string,
}) => any

// gets user shopping records with user id
type get_shopping_records = (_: {
user_id: string,
}) => any

} // namespace functions

# Instructions
Answer the following questions as best you can.
<|end|>

<|start|>user<|message|>Show me shopping records from Yi Tseng<|end|>
<|start|>assistant<|channel|>commentary to=functions.get_user_id<|constrain|>json<|message|>{"firstname": "Yi", "lastname": "Tseng"}<|call|>
<|start|>functions.get_user_id to=assistant<|message|>12345678<|end|>
<|start|>assistant<|channel|>commentary to=functions.get_shopping_records<|constrain|>json<|message|>{"user_id": "12345678"}<|call|>
<|start|>functions.get_shopping_records to=assistant<|message|>['apple', 'banana', 'cake']<|end|>
<|start|>assistant<|channel|>final<|message|>The shopping records from Yi Tseng are: apple, orange, and cake.<|end|>

In this case, you can find out there are also parts like initial prompt, CoT with tools use.

For these kind of models, you don’t have to instruct them to follow certain format to achive ReAct, as long as your program can keep providing the tool call result, LLMs will complete the task with ReAct automatically.

You can find the example code here, with gpt-4.1-mini model:

https://github.com/Yi-Tseng/medium-what-to-know-before-llm-fwk/tree/main/part3-react

Summary

You should be able to understand how ReAct works in the back scene. Many agentic AI uses this kind of technique to work on tasks, but somethimes frameworks can be unclear since it has multiple abstractions which allows you to develop agents without understanding how LLM works.

Next, I am going to talk about how a LLM application work with MCP servers, no abstration layers, just simple http/json-rpc calls 😃

Reference:

ReAct: Synergizing Reasoning and Acting in Language Models

Prompt Engineering Guide

The Illustrated GPT-OSS