Think Harder, Not Bigger: How OptiLLM Boosts LLM Accuracy Up to 10x at Inference Time Without Training

When an AI application is not performing well enough on reasoning tasks, the standard response follows a familiar pattern. The team either fine-tunes the existing model on domain-specific data, spending weeks of engineering time and substantial compute budget, or they swap to a larger, more capable model at a higher cost per token. Both approaches work. Both also carry meaningful costs in time, money, and operational complexity.

Fine-tuning requires curated training data, a training pipeline, evaluation infrastructure, and ongoing maintenance as the model degrades relative to newer base models over time. Upgrading to a larger model increases the API cost on every single call, not just the ones where the extra capability is genuinely needed, and it introduces dependencies on a specific provider and pricing tier that can change.

OptiLLM proposes a different approach entirely: instead of changing the model, change what happens at inference time. Rather than training the model to reason better, spend additional compute on each request to reason more thoroughly before returning an answer. The model stays the same. The API provider stays the same. The improvement comes from how the query is processed before it reaches the model and how the responses are selected, combined, or refined before they reach the application.

What OptiLLM Is

OptiLLM is an open-source inference proxy that implements more than 20 state-of-the-art reasoning optimization techniques in a single package. It exposes an OpenAI-compatible chat completions endpoint, which means any application or framework that already works with OpenAI’s API can route calls through OptiLLM without changing any client code except the base URL and, optionally, the model name prefix.

The proxy intercepts requests, applies the selected optimization technique, makes one or more calls to the underlying model, processes and synthesizes the results, and returns a final response that is typically more accurate than the raw model output would have been. From the application’s perspective, it called an LLM and received an answer. The OptiLLM layer between the application and the model handled everything else.

Installation takes a single command:

pip install optillm

Starting the server requires setting an API key and running one command:

export OPENAI_API_KEY="your-key-here"
optillm

The proxy starts on port 8000 by default and is immediately ready to receive requests. Switching an existing application to route through it requires changing one line: the base URL from the provider’s endpoint to the local OptiLLM server.

The Inference Time Scaling Insight

The fundamental insight behind OptiLLM is that model accuracy is not a fixed property. It is a function of how much compute is spent on a given request. This idea, which has come to be called inference time scaling or test-time compute scaling, has been demonstrated across a range of reasoning domains and model families.

A model that returns the first answer it generates achieves some baseline accuracy. The same model, given more compute to generate multiple candidate answers, verify them against each other, explore different reasoning paths, or apply formal logical verification, consistently achieves higher accuracy on tasks that require genuine reasoning. The relationship between additional inference compute and accuracy improvement is not linear across all tasks, but for mathematics, formal logic, code generation, and structured reasoning problems, the improvements can be substantial.

OptiLLM makes this insight practical by packaging the most effective inference time techniques into a single proxy that can be configured with a model name prefix. A developer who wants Mixture of Agents optimization adds moa- to the model name. A developer who wants Best of N sampling adds bon-. The technique is applied transparently without any changes to the application code.

The techniques can also be combined. The & operator chains them in sequence, with each technique receiving the output of the previous one. The | operator runs them in parallel and returns multiple candidates:

response = client.chat.completions.create(
  model="gpt-4o-mini",
  messages=[{"role": "user", "content": "your question here"}],
  extra_body={"optillm_approach": "bon|moa|mcts"}
)

The Optimization Techniques in Practice

OptiLLM bundles techniques that span a wide range of optimization strategies, from simple sampling approaches to complex multi-agent architectures and formal verification systems.

MARS (Multi-Agent Reasoning System) coordinates multiple agents with diverse temperature settings to explore the solution space from different angles. The agents cross-verify each other’s outputs and iteratively improve the response through multiple rounds of critique and refinement. On the AIME 2025 mathematical competition benchmark, MARS takes Gemini 2.5 Flash Lite from 43.3% to 73.3% accuracy, a gain of 30 points on one of the most challenging publicly available math benchmarks.

Mixture of Agents (MOA) generates multiple independent responses and uses critique rounds to synthesize a final answer that incorporates the best reasoning from each individual response. On the Arena-Hard-Auto benchmark, which evaluates response quality across a wide range of tasks, MOA applied to GPT-4o-mini produces results that match GPT-4’s performance. This is the clearest demonstration of what inference time scaling can accomplish: a smaller, cheaper model reaching the performance level of a larger one by spending more compute per request rather than using a larger model to begin with.

CePO (Cerebras Planning and Optimization) combines Best of N sampling, Chain-of-Thought reasoning, Self-Reflection, and Self-Improvement prompting into an integrated pipeline. Applied to Llama 3.3 70B on the Math-L5 benchmark, it produces an 18.6 point improvement from 51.0% to 69.6%. A companion technique called LongCePO extends this approach to long documents using a divide-and-conquer processing strategy, achieving a 13.6 point improvement on the InfiniteBench long-context benchmark.

Monte Carlo Tree Search (MCTS) applies the same algorithmic approach that achieved superhuman performance in strategic games to the problem of selecting optimal response paths in language model outputs. The model explores multiple reasoning branches, evaluates them based on expected outcome quality, and selects the path with the highest estimated value before committing to a final response.

PlanSearch implements a search algorithm over candidate plans expressed in natural language before executing any of them. By generating and evaluating multiple solution approaches before beginning implementation, the model avoids committing to an inferior strategy early. On the LiveCodeBench programming benchmark, PlanSearch applied to GPT-4o-mini produces a 20% improvement in pass@5 rate.

Z3 Theorem Prover routing is qualitatively different from the other techniques. Rather than improving probabilistic outputs through sampling or multi-agent approaches, it routes logical reasoning problems to the Z3 formal verification system. For problems that can be expressed as formal logical constraints, Z3 produces provably correct answers rather than statistically likely ones. The combination of an LLM for problem understanding and formulation with a theorem prover for verification is one of the most principled approaches to formal reasoning available in any inference proxy.

AutoThink classifies incoming queries by complexity and applies steering vectors to enhance reasoning for queries that require it. By calibrating the reasoning intensity to the actual complexity of the request, AutoThink avoids the token overhead of applying maximum reasoning effort to simple queries while ensuring that complex problems receive appropriate treatment.

Chain-of-Thought with Reflection adds explicit reflection and verification steps to the standard chain-of-thought reasoning pattern. The model generates its reasoning chain, then explicitly reflects on whether each step is correct, and produces a final output only after completing this verification loop.

Integration: Three Ways to Select a Technique

One of the practical strengths of OptiLLM is the flexibility of how optimization techniques are selected. Three independent mechanisms all work, and they can be used interchangeably depending on what fits the application architecture best.

The first approach is the model name prefix. Adding the technique slug before the model name tells OptiLLM which technique to apply for that request:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1")
# Mixture of Agents applied to GPT-4o-mini
response = client.chat.completions.create(
    model="moa-gpt-4o-mini",
    messages=[{"role": "user", "content": "Solve: If 2x + 3 = 7, what is x?"}]
)

The second approach is the extra_body parameter, which allows the technique to be specified separately from the model name. This is useful when the model name is set programmatically and cannot easily be prefixed:

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "your question here"}],
    extra_body={"optillm_approach": "cepo"}
)

The third approach is inline specification within the prompt itself, which requires no changes to the API call structure at all:

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{
        "role": "user",
        "content": "<optillm_approach>re2</optillm_approach> How many r's are there in strawberry?"
    }]
)

Multi-Provider Support and Local Inference

OptiLLM works with any OpenAI-compatible API endpoint, which through the LiteLLM integration effectively means it works with virtually any major AI provider. Google Gemini models, Anthropic Claude, Cerebras, Azure OpenAI, AWS Bedrock via proxy, and any self-hosted model that exposes an OpenAI-compatible endpoint are all accessible.

Using a non-OpenAI provider is as simple as setting the corresponding API key and using the LiteLLM model naming convention:

import os
os.environ['GEMINI_API_KEY'] = "your-gemini-key"
response = client.chat.completions.create(
    model="moa-gemini/gemini-1.5-flash-002",
    messages=[{"role": "user", "content": "your question here"}]
)

For teams that want to run inference entirely on local hardware, OptiLLM includes a built-in inference server that loads HuggingFace models directly. Multiple LoRA adapters can be layered on top of a base model with runtime selection:

response = client.chat.completions.create(
    model="meta-llama/Llama-3.2-1B-Instruct+patched-codes/Llama-3.2-1B-FastApply+patched-codes/Llama-3.2-1B-FixVulns",
    messages=messages,
    extra_body={"active_adapter": "patched-codes/Llama-3.2-1B-FastApply"}
)

The local inference server also unlocks decoding-level techniques that are not available when using external APIs, including CoT decoding, which elicits chain-of-thought reasoning without explicit prompting, and entropy decoding, which applies adaptive sampling based on the uncertainty of each generated token.

The Plugin Ecosystem

Beyond the core optimization techniques, OptiLLM includes a plugin system that extends its capabilities in directions that go beyond reasoning improvement.

The Memory plugin implements a short-term memory layer that enables effectively unbounded context length with any model by managing what is retained in the active context window across turns. The Privacy plugin anonymizes personally identifiable information in requests before they leave the local environment and restores the original values in the response, making it practical to use cloud AI APIs with data that would otherwise require on-premises processing.

The Deep Research plugin implements Test-Time Diffusion Deep Research, an iterative refinement approach for generating comprehensive research reports that draws on multiple rounds of search, synthesis, and revision.

The MCP (Model Context Protocol) client plugin connects OptiLLM to any MCP server, exposing filesystem access, database queries, web search, GitHub integration, and any other capability available through the MCP ecosystem directly to the language model. Both local servers running via standard input/output and remote servers accessible via Server-Sent Events or WebSocket connections are supported. Configuration is handled through a JSON file at ~/.optillm/mcp_config.json.

A straightforward MCP configuration connecting a local filesystem server and a remote GitHub server looks like this:

{
  "mcpServers": {
    "filesystem": {
      "transport": "stdio",
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-filesystem", "/home/user/documents"],
      "description": "Local file system access"
    },
    "github": {
      "transport": "sse",
      "url": "https://api.githubcopilot.com/mcp",
      "headers": {
        "Authorization": "Bearer ${GITHUB_TOKEN}"
      },
      "description": "GitHub repository and issue management"
    }
  }
}

The Proxy plugin adds load balancing and failover across multiple LLM providers with health monitoring and round-robin routing, which transforms OptiLLM from a single-provider optimization layer into a full multi-provider routing infrastructure.

Deployment and Security

OptiLLM binds to localhost by default, making it secure for development use without additional configuration. For production deployments where external connections are required, the --host 0.0.0.0 flag enables them, and an API key can be configured to require bearer token authentication on all requests:

optillm --host 0.0.0.0 --optillm-api-key your-secret-key

Docker deployment is supported with three image variants covering different use cases. The standard image includes all dependencies for local inference and plugins. The proxy-only image is a lightweight variant without local inference capabilities, appropriate for deployments that exclusively use external API providers. The offline image is a self-contained variant with pre-downloaded models for fully air-gapped environments.

SSL configuration supports both custom CA certificates for environments with corporate proxies and, for development only, the option to disable verification entirely. The proxy logs all requests with technique selection information, making it straightforward to monitor which optimizations are being applied and at what rate.

When to Use Which Technique

Not every optimization technique is appropriate for every task. The proxy’s auto mode attempts to select the best approach based on the incoming request, but understanding the tradeoffs helps in making deliberate choices.

For mathematical reasoning and formal problem-solving, MARS and MCTS tend to produce the largest absolute gains because these problems have verifiable correct answers that benefit from exploration and cross-verification. The Z3 integration is particularly powerful for problems that can be expressed as logical constraints, where the formal verification guarantees rather than estimates correctness.

For code generation, PlanSearch and CePO both show strong results because exploring multiple solution strategies before committing to one avoids the common failure mode of optimizing a locally plausible but globally suboptimal approach. Best of N sampling is a simpler alternative that often captures a meaningful fraction of the improvement at lower compute cost.

For general reasoning tasks where the problem type is variable, Mixture of Agents provides consistent improvement across diverse domains because the cross-critique mechanism catches errors that any single reasoning path might miss. The ArenaHard-Auto results showing GPT-4o-mini matching GPT-4 reflect this generality.

For long-document tasks, LongCePO’s divide-and-conquer approach addresses the specific failure mode of reasoning degradation on extended inputs, where models lose coherence across the full document length.

Conclusion

OptiLLM makes a compelling practical case for inference time scaling as a first-line tool for improving LLM accuracy before reaching for fine-tuning or model upgrades. The combination of 20-plus optimization techniques, a drop-in OpenAI-compatible interface, multi-provider support through LiteLLM, local inference capability, and a plugin ecosystem that extends into memory management, privacy, research, and MCP tooling makes it one of the most complete inference optimization solutions available in open source.

The benchmark results are real and reproducible. A 30-point accuracy improvement on AIME 2025, an 18.6-point gain on Math-L5, and GPT-4o-mini reaching GPT-4’s performance on Arena-Hard-Auto are not marginal improvements. They represent the kind of capability shift that previously required either larger models or fine-tuning, achieved instead by thinking more carefully at inference time with the same model the application was already using.

For any team running LLM-powered applications where reasoning accuracy matters, OptiLLM is worth deploying as an evaluation step before any decision to invest in fine-tuning or model upgrades. The setup investment is measured in minutes. The potential accuracy gains are measured in double-digit percentage points.

The repository is available at: https://github.com/algorithmicsuperintelligence/optillm