Have o1 Models Solved Human Reasoning?

Image Generated By ChatGPT

OpenAI made waves in the AI community with the release of their o1 models. As the excitement settles, I feel it’s the perfect time to share my thoughts on LLMs’ reasoning abilities, especially as someone who has spent a significant portion of my research exploring their capabilities in compositional reasoning tasks. This also serves as an opportunity to address the many “**Faith and Fate**” questions and concerns I’ve been receiving over the past year, such as: Do LLMs truly reason? Have we achieved AGI? Can they really not solve simple arithmetic problems?

The buzz around the o1 models, code-named “strawberry,” has been growing since August, fueled by rumors and media speculation. Last Thursday, Twitter lit up with OpenAI employees celebrating o1’s performance boost on several reasoning tasks. The media further fueled the excitement with headlines claiming that “human-like reasoning” is essentially a solved problem in LLMs.

Without a doubt, o1 is exceptionally powerful and distinct from any other models. It’s an incredible achievement by OpenAI to release these models, and it’s astonishing to witness the significant jump in Elo scores on ChatBotArena compared to the incremental improvements from other major players. ChatBotArena continues to be the leading platform for assessing models in live, real-world tasks where companies cannot easily manipulate the system.

To those outside the research community, it might appear as though a revolutionary new paradigm has suddenly emerged from OpenAI’s labs. However, it’s important to provide context around the “o1 models” and reasoning in general, explaining how foundational work by numerous researchers (including OpenAI employees) has contributed to such development.

My goal in this blog is to take you beneath the surface of o1 and uncover its magic. I will explain that while we’re making significant strides in LLM reasoning, we’re not quite there yet. The problem is far from solved.

[Disclaimer: I don’t know in reality how o1 model works, this is just my speculation]

Is o1 a magical model: Deciphering o1 training

We still don’t know the exact details of the inner workings of o1 due to the high level of secrecy, but as a researcher who has been working on LLMs and reasoning, I can make an educated guess about how it might function. It’s unclear whether o1 is a single model or a set of models; there are various views and speculations on this. However, I suspect the core concept revolves around a unified system. The lead on o1 has confirmed that it is one model, although the specifics of its architecture remain undisclosed.

According to OpenAI’s blog post, these models are trained using reinforcement learning (RL) with chain of thought (CoT) to reason through problems step by step before generating the final answer. Successfully implementing RL at such a large scale is undoubtedly a significant achievement and an impressive feat of engineering.

Our large-scale reinforcement learning algorithm teaches the model how to think productively using its chain of thought in a highly data-efficient training process. We have found that the performance of o1 consistently improves with more reinforcement learning (train-time compute) and with more time spent thinking (test-time compute). The constraints on scaling this approach differ significantly from those of LLM pretraining, and we are continuing to investigate them.

Speculated Training Process

How was o1 trained? It’s unclear whether it involved a sophisticated, newly developed RL algorithm that explicitly trained the model to backtrack if it made a mistake, or if it follows more common, standard techniques. What seems evident to me is that o1 likely follows the successful alignment pipeline that has become standard in the field. This process typically begins with Supervised Fine-Tuning (SFT) on CoT data, followed by scoring the generated reasoning chains using a process reward model. The resulting score is then used to optimize the model, enhancing its ability to think through solutions. The optimization likely employs policy gradient algorithms like Proximal Policy Optimization (PPO). PPO generates online completions, and to produce these completions, OpenAI must have explored various reasoning paths in search of optimal trajectories. I will discuss below several techniques that may have been used.

The key difference between o1’s training and that of previous models (GPT-4, ChatGPT, GPT-4-o) lies primarily in the nature of the training data and the focus on rewarding the intermediate steps that lead to the solution. In previous iterations, models were mostly trained on (prompt, solution) pairs: given a problem as input, the model was expected to generate the final answer as output. In contrast, o1 is trained on input problems paired with detailed, step-by-step explanations of how to reach the solution, with each step being scored.

Data Collection: OpenAI likely paid highly skilled annotators to create complex reasoning paths, possibly including multiple paths for a single problem. They probably also collected incorrect chain of thought paths that are difficult to identify as wrong, creating contrastive sets to strengthen the learning signal. In addition to human data, they likely collected large-scale synthetic data seeded by human-created content. Automating this process is challenging, as a single mistake in a step can lead to exponential error propagation, so the synthetic data must have been carefully audited and filtered to ensure quality. Obtaining this high-quality data is crucial for building o1. OpenAI is clearly aware of the sensitivity of this information, which is why they have chosen not to display the reasoning trace to users and have made efforts to prevent attempts to extract this data from the model through jailbreaking.

Inference-time generation

The real innovation in these models seems to lie in the inference stage, which requires significant computational resources and engineering. The o1 models appear to be the first product that enables large-scale text search in real-time, representing a major breakthrough that is set to transform deployment frameworks and raise expectations for AI products. The improvements we’re seeing confirm the existence of inference-time scaling laws. This means that the number of problems solved by any attempt scales with the number of samples generated from the model at inference time. Consider this: the raw GPT-4 might generate only 1 correct response out of 1000 samples, a success rate of just 0.1%. By using a reward model to return the highest-scoring answer to the user, the success rate skyrockets to 100%. But the question remains: is this a real improvement, or just clever filtering?

Google released a paper on inference-time scaling laws in late July. Their experiments reveal a clear relationship between finding the correct answer and the number of generated answers from LLMs. This relationship is often log-linear and can be modeled with an exponentiated power law.

How does o1 work at inference time? During inference, o1 generates multiple chains of thought (what OpenAI refers to as the model “thinking”). It then searches for the best answer using a scoring function to evaluate these different reasoning trajectories. This process likely involves generating thousands, if not hundreds of thousands, of potential reasoning paths, distributing this generation across many GPUs, and scoring and selecting the most promising solutions. The scale of this inference-time computation is unprecedented, potentially involving hundreds or thousands of GPUs working in parallel to explore the vast space of possible reasoning paths. The number of chain-of-thought steps generated at inference time is currently set by OpenAI, and users cannot control it. This has significant implications for the response time essentially, how long one must wait for an answer, or “how much the model is thinking.” OpenAI has indicated that in the future, they plan to give users more control over this waiting time.

Jim Fan summarized nicely in a Twitter post the shifting compute expenditures for next-generation AI systems.

So what kind of inference algorithms OpenAI may have used?

Inference Time Decoding Algorithm

The most straightforward way to generate text at inference time is to use greedy sampling or nucleus sampling combined with top-k sampling, without imposing any constraints on the generation. However, these techniques are suboptimal and don’t necessarily lead to the best outcome. They can be inefficient, as there’s no point in generating a path if it’s doomed to be poor from the very first incorrect steps. This is where decoding time algorithms come into play. Controlled decoding has seen significant success in the NLP community, as it allows for improving model performance without going through intensive training. Examples include Neurologic decoding (Lu et al., 2020), and GBS (Hokamp and Liu, 2017), which generalize beam search for lexically constrained decoding, DExperts (Liu et al., 2021b), and more. OpenAI has likely used a reinforcement decoding algorithm similar to Tree of Thoughts, which I will explain below.

Tree of Thoughts:

(Yao et al., 2023) introduced this approach last year. It uses LLM self-feedback: the model samples one thought at a time leading to the solution, then self-evaluates by asking whether the generated step is likely correct. If deemed incorrect, the model doesn’t proceed with that path. By evaluating multiple future reasoning steps before continuing, the model has opportunities to identify and correct errors. This approach aligns with key successes in deep reinforcement learning, such as AlphaGo. The process involves complex decoding algorithms that balance exploration of different thought paths with exploitation of promising directions. OpenAI, at least during training, likely has a mechanism to adjust the amount of compute used through search depth, width, or heuristic calculation (scoring the nodes of the search).

Inference Time Scoring functions

Scoring in LLMs via Consensus : is the process of generating multiple candidates and selecting the most common one using exact matching, such as regex. A straightforward way to pick the best response from many candidates is through majority voting, meaning selecting paths that lead to the same final solution. Consensus is easy to implement when returning a single number. However, it becomes challenging when writing a proof, as it’s unlikely that the model will generate the same proof multiple times. By applying this consensus method over 1000 samples, Minerva from Google Research improved its performance on the MATH dataset from 33.6% to 50.3%.

Scoring in LLMs using Best-of-N: is a method where multiple solutions are sampled and then scored using a reward model. The solution with the highest score is returned. With a sufficiently accurate reward model, Best-of-N can outperform consensus methods. However, this approach is ultimately limited by the quality of the reward model and risks overfitting to errors. In this context, the reward model is often referred to as the “outcome” model.

Scoring using Process Reward Models: involves verifying each step individually rather than just the final solution. This method is more effective than outcome reward models (which score only the final solution and not the steps leading up to it) and majority voting when searching over a large number of model-generated solutions. PRMs provide a richer signal because they specify both how many of the first steps were correct and the precise location of any incorrect steps. However, automating process supervision is not straightforward; it requires human data labelers to label the correctness of each step in model-generated solutions. To reduce reliance on costly human feedback, large-scale models are used to supervise the training of smaller models. o1 likely employed PRMs to score the generated chains of thought both during training and at inference time. The image below from @prnvrdy offers a nice illustration.

Have LLMs solved reasoning?

Ok, having discussed how o1 might roughly be working, let’s reflect on whether we have truly cracked reasoning and whether CoT and inference-time sampling mean reasoning is solved. Discussions about reasoning have been heated since the emergence of LLMs, and this debate isn’t new with the appearance of ChatGPT. Even with the release of BERT and GPT-3, the research community was already questioning whether these LLMs could truly reason. One famous debate took place between Geoffrey Hinton, Yoshua Bengio, and Yann LeCun, who disagreed on whether LLMs can understand what they say. Geoffrey Hinton and Yoshua Bengio have warned about existential risks, while Yann LeCun argues that current LLMs haven’t yet reached dog-level intelligence.

But this hype intensified even further with the release of ChatGPT, GPT-4, Gemini, Claude, LLaMA 3, and others. The marketing of these products, along with their convincing-sounding responses (even when incorrect), has painted an overly optimistic picture, leading many people to believe that these models can handle any task effortlessly.

This led me to question their true capabilities: have these models truly cracked reasoning? Can they handle any compositional task? Can they learn implicit problem-solving rules through chain-of-thought training? And under what conditions do they succeed, fail, and why?

I still don’t have answers to all of these questions, and it’s difficult to give a final verdict on their reasoning skills. The fact that they are already able to answer very complex queries is mind-boggling in itself. But the mystery lies in how they do it: whether they’ve truly learned to reason like humans, or if their reasoning is a mixture of pattern recognition combined with some form of reasoning.

My investigations led to the “Faith and Fate” paper, where we studied three compositional reasoning tasks: multiplication, dynamic programming, and one NLP task called the Einstein puzzle. The beauty of these tasks lies in their compositional nature, meaning we can easily adjust their complexity and observe how this affects LLM behavior (e.g., comparing 2 x 2 multiplication to 10 x 10 multiplication).

Chain-of-Thought Training in LLMs: Performance Boost and Limitations

TLDR: Chain of thought training leads to a significant boost in performance, BUT performance drops to 0 as the complexity of the task increases and out-of-distribution (OOD) cases are encountered.

We were particularly curious about whether training these LLMs with a massive amount of (prompt, chain-of-thought, solution) in an SFT style would enable them to learn the underlying algorithm and achieve 100% accuracy, regardless of the task’s complexity. Could they, for example, generalize from 5x5 multiplication training to solve 6x6 multiplication problems at inference time?

Across all tasks, we observed a performance boost compared to just (prompt, solution) training. The main issue, however, was the OOD performance. The performance quickly drops to 0% on tasks like 5-digit x 5-digit multiplication if the model was trained up to 4x4. The “o1-preview” model fails with long-digit multiplication, and the “o1” models incorrectly answer an 8x8 digit question.

Prompting “Can you tell me how much 96745982 times 93678239 ?” The correct answer is “9,062,993,224,085,698”

Yuntian Deng tested “o1-mini” on multiplication tasks up to 20x20 — it solves up to 9x9 multiplication with decent accuracy, while GPT-4o struggles beyond 4x4. GPT-4o was likely not trained with the same amount of CoT trajectories, leading to its lower performance.

Let’s go back to the question: “Can you tell me how much 96745982 times 93678239 ?” The correct answer is “9,062,993,224,085,698” o1’s answer is “9,066,591,470,043,979,798”

If you look closely at the “o1” answer, you can notice that the model partially predicted the response correctly (the first and last 2 digits), even though the overall answer is incorrect. This same phenomenon is observed in “Faith and Fate,” and we explain it through relative information gain. What’s happening during model training is that the model is learning patterns and associations in the data, but it may struggle with more complex or out-of-distribution examples, leading to partial but not fully correct answers. This suggests that while the model has learned certain parts of the task, it hasn’t generalized effectively to the entire problem, especially in cases that require higher complexity or more steps in the reasoning process.

The next-token prediction leads the model to learn superficial, spurious patterns. This means that if an output element heavily relies on a single or a small set of input features, LLMs are likely to recognize such correlations during training and make partial guesses during testing without executing the full multi-step reasoning required by the task. In other words, the model learns shortcuts from the CoT training. There’s nothing wrong with learning shortcuts, as humans commonly use them to quickly deliver answers. However, the key difference is our ability to discern when and how to use shortcuts — a skill that LLMs seem to lack. Teaching LLMs with CoT and RL search does not necessarily resolve this issue, especially when the complexity of the task is extremely high.

LLMs Reduce Multi-Step Compositional Reasoning into Subgraph Matching

To understand more about why GPT-4 and ChatGPT cannot solve complex reasoning tasks with 100% accuracy, we tried to examine the success cases and determine whether they were due to genuine reasoning or simply because the models had been exposed to similar training examples during their training.

How do we do this? We formulate these tasks as computation graphs, which break down problem-solving into smaller, functional steps.

For each graph, we computed the average frequency of partial computations that appear in the training data needed to solve a task, for both correctly and incorrectly predicted examples. We found that Transformers’ successes are heavily linked to having seen significant portions of the required computation graph during training — suggesting that compositional behaviors may indeed be pattern matching.

Although the final prediction may be highly compositional, the true compositionality is much less impressive because the solutions could easily be extracted from the input-output sequences present in the training data. This type of learning can be very effective when the compositional complexity of tasks is low, but it becomes less efficient as tasks increase in complexity. Shortcut learning via pattern-matching may lead to fast correct answers when similar compositional patterns are available during training, but it does not allow for robust generalization to uncommon or complex examples. Despite RL-training and search, it’s very likely that o1 exhibits the same behavior, as it produces similar patterns.

As research progresses, we may need to redefine our concepts of “understanding” and “reasoning.” Perhaps these models are developing forms of cognition that are fundamentally different from human reasoning.

The Inference-Time Algorithm: A Step Forward or a Brute Force Detour?

While inference-time algorithms undeniably enhance LLM performance, we must question whether this approach truly advances our quest for artificial reasoning. Are we perhaps overlooking more clever, human-like solutions in our rush to improve benchmark scores?

OpenAI claims that models like o1 now “think” like humans. But let’s be clear: generating thousands, hundreds of thousands, or even millions of trajectories and then scoring them using a scoring function is far from human cognition. It’s a brute-force approach that, while undeniably effective, is fundamentally different from human reasoning.

CoT combined with RL search and reward modeling scoring is incredibly powerful, but do we really need to search over thousands or millions of chains of thought to answer math questions that humans can solve with a simple algorithm or basic tools like a calculator? Maybe, once compute becomes cheaper, it wouldn’t be a big deal — especially since OpenAI has demonstrated that they found a way to scale it up, and scale seems to be the most effective method, outperforming compute-poor models by a large margin. This is reminiscent of the bitter lesson from Rich Sutton.

The biggest lesson that can be drawn from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin. The ultimate reason for this is Moore’s law, or more specifically, its generalization of the continued exponential decrease in the cost per unit of computation.

But for open-ended tasks (e.g., writing, medicine), where solutions are rarely binary and often highly contextual, these brute-force methods won’t necessarily work. Search and RL can be highly effective in domains with clear success metrics (e.g., coding or mathematics), but when the “correctness” of an answer becomes ambiguous or subjective, they become an inefficient use of time and resources.

Why Does the Debate Over Whether LLMs Show Signs of Reasoning or Not Matter?

It matters at least for me as a researcher. But, …

From a functional standpoint, it doesn’t really matter. The broad set of users primarily care about whether LLMs effectively satisfy their requests. Whether reasoning is involved or the underlying process of how it arrived at a solution may be less relevant. The focus shifts to the end result rather than the specific cognitive steps taken.

However, from a deeper understanding and interpretability standpoint, we need to understand the full picture of how they function. This knowledge is crucial to help us make informed decisions regarding their architecture and training strategies, ultimately advancing their capabilities.

There’s a big problem in doing this: how tricky their evaluation is. We’re currently testing these models based on human-like mental abilities, designing benchmarks, and testing whether they can follow the same algorithms humans use. But what if they have a completely different computational graph or fundamentally different mechanisms from humans? Are our evaluations fair or complete?

The divergence in thinking processes becomes more significant here. If an LLM uses fundamentally different mechanisms or lacks a clear explanatory framework, understanding how it arrived at the output becomes incredibly challenging. Without a clear insight into their reasoning process, we’re left questioning whether our current evaluation methods are truly adequate or if they’re potentially overlooking crucial aspects of AI capabilities and limitations.

Cognition Limit: Is There Some Upper Limit On How Much Cognition LLMs Can Obtain From Language Alone?

This is the most asked question: I believe that to achieve a deeper comprehension of reality, we need systems that learn from sensory inputs like video, which allow them to grasp how the world truly functions. Current LLMs are purely trained on massive amounts of text. However, I think most human knowledge is not tied to language, meaning that part of the human experience is not captured by AI.

It’s ironic that LLMs can now pass the Bar in the U.S., an examination required to become an attorney. However, they still struggle with basic tasks like simple multiplication or addition. Models that incorporate vision-based learning would provide a more profound understanding. There are many efforts underway to build multimodal systems that process text, audio, and video simultaneously, but are we there yet?

I think that without learning world models from sensory inputs and incorporating architectures capable of reasoning and planning (rather than just auto-regression), it may be difficult to achieve AI at the level of humans.

Embracing Agent LLMs as the Next Frontier

LLMs are undeniably powerful, but by design, they are limited to producing natural language, which prevents them from interacting with the real world or using external tools in the same way humans do. For example, we don’t need a gigantic LLM trained on millions of arithmetic problems to perform simple multiplication tasks. The model could instead directly call an external calculator.

Tool use has been gaining increasing popularity in the AI community recently. Sam Altman mentioned last week that OpenAI is actively working in this direction. The next step is to augment these models with planning modules that will enable them to make more strategic decisions about when to rely on their parametric knowledge and when to leverage external tools. These composite systems will improve the processing of long contexts, facilitate the orchestration of multiple tools, and enhance multi-turn conversational abilities, ultimately enabling more effective use of expert human feedback.

I will be writing more about LLMs and agents in the near future … Stay tuned!