Test-Time Compute: Scaling AI at Inference

Training larger models on more data was the standard path to better AI performance. That held true for a while, then returns began to diminish. High-quality data is finite, and the cost of large training runs has become prohibitive. Researchers needed a different approach. The shift now is toward allocating more computation during inference — the moment a model responds to a query — rather than during training. This is test-time compute (TTC).

This is the 4th article of our AI Engineering Series.

What Changes

A language model has two distinct phases: training and inference. Training is a one-time process. Inference is what happens every time you send a query. Traditionally, inference was a single forward pass — fast and cheap.

TTC changes inference. Instead of generating one immediate response, the model is given more computation at query time: more reasoning steps, multiple candidate answers, or iterative self-correction. The core finding, confirmed by Snell et al. [1], is that a small model given extra inference compute can outperform a model fourteen times its size running a single pass.

Why Models Can Generate Multiple Different Answers

This is worth understanding precisely, because it is central to how TTC works.

A language model does not retrieve stored answers. At every token (word) it generates, it produces a probability distribution over the entire vocabulary — every possible next word gets a score. The model then samples from that distribution. This sampling is stochastic: controlled by a parameter called temperature, it introduces randomness so the same prompt does not produce the same output every time.

Ask the model the same question twenty times, and you get twenty independent sampling runs. Each run takes a different path through the probability space, because the model learned many different human reasoning approaches during training — not one fixed method. This is what makes generating diverse candidate answers possible and meaningful.

The Main Techniques

Chain of Thought (CoT). The model generates intermediate reasoning steps before committing to a final answer [2]. This forces it to work through a problem sequentially, catching errors that a direct answer would skip. Basic CoT can be triggered by prompting; advanced versions involve reinforcement learning during training specifically to produce long, reliable reasoning traces.

Best-of-N Sampling. The model generates N independent answers to the same query. A verifier then selects the best one. Majority voting is the simplest verifier: whichever answer appears most often among N runs wins. Applying majority voting to DeepSeek-R1-Zero on the AIME 2024 benchmark raised its score from 71.0% to 86.7% [3] — no model change, just repeated sampling.

Beam Search. Rather than completing N full answers, beam search keeps the top-K partial answers alive at each generation step, pruning weaker paths continuously. When guided by a reward model, it focuses compute on the most promising reasoning branches [1].

Monte Carlo Tree Search (MCTS). The model explores a reasoning tree: it can try a branch, evaluate how promising it is, backtrack, and try another. Borrowed from game-playing AI, it suits complex multi-step planning tasks.

How the Best Answer Gets Chosen

Generating multiple candidates is only half of TTC. Selection matters equally.

Outcome Reward Model (ORM). A separately trained model scores only the final answer — correct or not. Fast and inexpensive, but it cannot catch errors in the middle of a reasoning chain.

Process Reward Model (PRM). Scores each individual reasoning step. If step 3 in a chain is wrong, the PRM flags it even when the final answer looks correct. Lightman et al. [4] showed that process supervision significantly outperforms outcome supervision on the MATH benchmark: their PRM solved 78% of a representative test subset. Step-level error rates dropped from 14.0% to 3.4% compared to ORM.

Code execution. For programming tasks, the answer is run directly. Passing tests confirms correctness — no learned judge needed.

What It Produced

OpenAI’s o1 (September 2024) was the first major public model built on TTC. On the American Invitational Mathematics Examination (AIME), GPT-4 scores roughly 12%. o1 scored 74% [5]. The underlying mechanism: the model generates extended chain-of-thought traces and uses reinforcement learning trained to reward correct reasoning paths.

DeepSeek-R1 (January 2025) reached comparable performance using a pure reinforcement learning approach, without human-annotated chain-of-thought data [3]. It also introduced dynamic compute allocation: more tokens for hard problems, fewer for easy ones.

OpenAI o3 (2025) extended this further, reaching 96.7% on AIME and 87.5% on ARC-AGI — a benchmark previously considered very difficult for AI systems [5].

The Cost Trade-Off

More inference compute means higher cost per query and higher latency. Running o3 at maximum compute settings has been reported at roughly $3,000–$3,500 per query. Lighter variants cost fractions of a cent. Snell et al. [1] confirmed there is no universally optimal TTC strategy: at low compute budgets, shorter responses win; at medium budgets, beam search is optimal; at high budgets, majority voting over many samples performs best.

The practical implication: TTC should be applied selectively. Routine queries do not need it. Complex reasoning tasks — mathematics, code generation, multi-step planning — benefit significantly.

Where This Fits in the Scaling Picture

Pre-training scaling — larger models, more data, more training compute — has been the dominant paradigm since at least 2020 [6]. It is showing diminishing returns. TTC introduces a second scaling axis: inference compute can now be traded directly for output quality, per query, at runtime. This does not replace pre-training; it supplements it. The two levers operate independently, and their combination defines the current frontier of model capability.

References

[1] C. Snell, J. Lee, K. Xu, and A. Kumar, “Scaling LLM test-time compute optimally can be more effective than scaling model parameters,” arXiv preprint arXiv:2408.03314, 2024.

[2] J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” in Proc. NeurIPS, vol. 35, pp. 24824–24837, 2022.

[3] DeepSeek-AI, D. Guo, D. Yang, H. Zhang et al., “DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning,” arXiv preprint arXiv:2501.12948, 2025.

[4] H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe, “Let’s verify step by step,” in Proc. ICLR, 2024. [Online]. Available: https://arxiv.org/abs/2305.20050

[5] OpenAI, “Learning to reason with LLMs,” OpenAI Technical Blog, Sept. 2024. [Online]. Available: https://openai.com/index/learning-to-reason-with-llms/

[6] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan et al., “Language models are few-shot learners,” in Proc. NeurIPS, vol. 33, pp. 1877–1901, 2020.