🚀 A New RL Trick Beats GRPO

Photo by Vitaly Gariev on Unsplash

Reinforcement learning papers are appearing almost every week. Most of them introduce a slightly different loss function, a new clipping method, or another optimization tweak.

For Non-Members: Read here!

We can question these changes by asking: Is this actually changing how we train the models, or is it just another small improvement?

For a while, GRPO (Group Relative Policy Optimization) has been one of the most popular choices. It helped train reasoning models without requiring a value model, making RL training simpler and cheaper.

But the moment models started acting like agents instead of chatbots, many of those older assumptions stopped working.

Today’s models don’t just answer questions; they can browse websites, write code, call tools, fix bugs, and spend minutes solving a single task. Suddenly, the old RL pipelines begin to show their limits.

A recent paper introduces Single-rollout Asynchronous Optimization (SAO). The idea doesn’t make GRPO better, but rather asks a much bigger question: What if the problem isn’t GRPO itself, but the way we’re collecting rollouts?

The answer is surprisingly simple: stop waiting for groups of rollouts, train on one rollout as soon as it finishes, and redesign the RL pipeline around that idea.

Let’s understand why.

The problem with today’s RL training

Most reinforcement learning pipelines for LLMs work in batches. Imagine you give a model eight different prompts.

• Some prompts finish in 10 seconds.
• Others need Python tools, multiple reasoning steps, or hundreds of tokens and may take two minutes.
• Even if seven tasks are already done, the trainer waits for the slowest one before updating the model.

This is called synchronous training.

For normal text generation, this isn’t a huge issue, but for AI agents, it’s really painful.

Agent trajectories are unpredictable. One coding problem may finish quickly, while another spends hundreds of turns debugging the code. As models become more agent-like, waiting for the slowest rollout wastes a lot of GPU time.

Why asynchronous RL sounds like the answer

A natural idea is asynchronous training. Instead of waiting for an entire batch, why not update the model whenever a rollout is ready?

Imagine eight students taking an exam. Seven submit their papers early, but the teacher refuses to start grading until the last student finishes. That’s essentially what synchronous RL does.

Asynchronous RL leads to better GPU utilization, making the training faster.

Unfortunately, a new problem appears. While one rollout is being generated, the model may already receive several updates from other rollouts. By the time that rollout finishes, it was generated by an older version of the model.

This creates what RL researchers call policy lag or off-policy training. Now you’re updating today’s model using data generated yesterday, and too much policy lag makes the training unstable.

GRPO makes this even harder

GRPO works by generating multiple responses for the same prompt. Instead of judging one answer, it compares a group of answers. The model learns which response performed better than the others. This works nicely in synchronous training.

But imagine asynchronous training.

• One response finishes in 15 seconds.
• Another finishes after 90 seconds.

Training cannot begin until the whole group is ready. So even though asynchronous training wants immediate updates, GRPO forces everything to wait.

The two ideas simply don’t fit together very well.

SAO changes one simple thing

Here’s the idea that caught my attention.

Instead of generating a group of responses for every prompt, it generates just one rollout.

One prompt → One trajectory → One update

As soon as the rollout finishes, the training starts immediately.

• No waiting.
• No group synchronization.
• No slowest sample holding everyone back.

At first glance, this sounds risky. After all, GRPO uses multiple samples, and comparing them reduces the variance.

Wouldn’t single-rollout RL become noisy? Yes. Unless you improve the rest of the training pipeline. That is exactly what SAO does.

Why single-rollout training actually works

The paper isn’t just about using one rollout. It introduces several ideas that make single-rollout training actually work.

1. Better handling of policy lag

Traditional PPO-style training keeps multiple policy versions around: current, old, and rollout policy. And tracking all of these becomes difficult in asynchronous systems because rollouts may span multiple model updates.

SAO removes this complexity.

Instead of comparing against an old stored policy, it directly uses the log probabilities recorded during rollout generation. Then it applies a strict token-level trust region.

If a token differs too much from the rollout policy, SAO ignores it during optimization. Instead of forcing unstable updates, it completely masks those tokens.

I liked this design because it removes a lot of bookkeeping without making the optimization process more complicated.

2. A stronger value model

Moving away from GRPO means you now depend on a value model again. If the value estimates are poor, then the training becomes noisy.

The authors noticed something interesting: the value model should learn faster than the policy. So instead of updating both equally, they update the critic twice for every policy update.

This helps the value estimates catch up with the changing policy. They also freeze the attention layers while training the value model.

Only the selected parts continue learning, and this keeps the gradients much more stable during long RL runs.

3. Skip observation tokens

Agent trajectories are different from normal text generation. A model writes an action, and the environment replies.

Then the model acts again. Those environment responses were never generated by the model. So why should they influence token-level advantage estimation?

SAO introduces a clever Skip-Observation GAE.

Instead of propagating value through environment tokens, it directly connects one model action to the next. The value function focuses only on the decisions made by the model, thus reducing the unnecessary noise during the RL training.

Does it actually beat GRPO?

SAO vs. GRPO

The short answer is yes, and the improvements aren't limited to a single benchmark; SAO consistently outperformed GRPO on multiple benchmarks:

• On AIME 2025, accuracy improved from 84.2% with GRPO to 97.3%.
• On BeyondAIME, performance increased from 54.8% to 74.8%.
• On IMOAnswerBench, accuracy rose from 55.8% to 74.0%.

The coding benchmark SWE-Bench Verified also improved from 27.0% to 29.8%. Perhaps even more important than the final scores is the training behavior.

Vanilla GRPO often became unstable after around 160 training steps. But SAO continued training stably for roughly one thousand steps while maintaining better performance.

Why this matters beyond benchmarks

Most recent RL improvements tweak clipping functions, reward normalization, or loss equations.

SAO asks a more practical question: How should RL work when models become long-running agents?

If you’ve ever watched an agent solve a coding task, you’ve probably noticed that most of the time isn’t spent generating text. It’s running tools, waiting for outputs, trying another approach, and repeating the process. During that time, other rollouts may have already finished. Waiting for every trajectory to complete before training starts begins to feel wasteful. That’s exactly the situation SAO is trying to fix.

The paper even demonstrates this in a simulated online learning setup where reward preferences change over time. Because SAO learns from one rollout at a time, it adapts much faster than methods that rely on historical reward averages.

Final thoughts

GRPO has played an important role in making reinforcement learning practical for reasoning models. It simplified training and removed the need for maintaining a separate value model in many settings.

But as AI moves toward agentic systems, new problems appear.

• Long rollouts.
• Variable execution time.
• Tool use.
• Changing environments.

SAO shows that solving these problems may require changing more than just the loss function. By replacing group rollouts with single rollouts, improving value estimation, and making asynchronous training stable, it offers a practical direction for the next generation of RL pipelines.

Whether SAO becomes the new standard remains to be seen. The paper itself notes that its results are based on large agentic reasoning and coding models, and it still depends on a well-trained value model and rollout log probabilities.

Still, the core idea is refreshingly simple.

We often think better RL comes from more complex objectives or more sophisticated reward functions. SAO reminds us that sometimes the bottleneck isn’t the optimization algorithm at all. Sometimes it’s simply the fact that we’re making fast rollouts wait for slow ones.

Digital Products

ML Interview Book: Crack Your Next ML Interview with Machine Learning Interview Playbook

Productivity Tool: ***Social Media Time Tracker: Take Back Your Time, a tool that annoys you when you log in to social media sites. Chrome Extension.***

Connect with the author

LinkedIn | YouTube | Threads | Instagram | Facebook

References

https://arxiv.org/pdf/2607.07508