Why Agentic RL Breaks (and How rStar2-Agent Fixes It) — Paper Review

https://arxiv.org/abs/2508.20722

If you’ve ever watched an LLM use tools during reasoning, you’ve seen the magic:

• It writes Python to verify steps.
• It catches arithmetic mistakes.
• It iterates toward better answers.

But there’s a hidden problem most people miss: tool environments are noisy.

A code call might fail with a syntax error, timeout, or malformed response — yet the model can still get the final answer right by luck. If your reward is only based on final correctness, training will reward these messy trajectories too.

That is the core insight behind rStar2-Agent: don’t complicate reward design — clean the rollout data instead.

The paper in one line

The Microsoft Research report introduces GRPO-RoC (Resample-on-Correct), a trajectory selection strategy for agentic RL that filters noisy positive trajectories while preserving diverse negative ones.

🔗 Paper: rStar2-Agent: Agentic Reasoning Technical Report https://arxiv.org/abs/2508.20722

The failure mode of vanilla GRPO

In a standard GRPO setup:

1. Sample a group of rollouts.
2. Compute group-normalized advantages.
3. Update using all trajectories.

The issue in agentic settings is subtle but severe:

• Two trajectories can both receive reward = 1.
• One is clean and tool-reliable.
• The other is noisy (tool errors, formatting issues), but still ends with a lucky-correct answer.

Vanilla GRPO treats them similarly. Over time, this can reinforce brittle tool-use behavior and lead to plateaus.The failure mode of vanilla GRPO

In a standard GRPO setup:

1. Sample a group of rollouts.
2. Compute group-normalized advantages.
3. Update using all trajectories.

The issue in agentic settings is subtle but severe:

• Two trajectories can both receive reward = 1.
• One is clean and tool-reliable.
• The other is noisy (tool errors, formatting issues), but still ends with a lucky-correct answer.

Vanilla GRPO treats them similarly. Over time, this can reinforce brittle tool-use behavior and lead to plateaus.

The GRPO-RoC idea (simple and clever)

GRPO-RoC changes sampling, not the reward function:

• Oversample 2G rollouts.
• Split into positive (reward=1) and negative (reward=0) sets.
• For positives, score quality with a penalty:

p_err = tool error rate

p_format = formatting violation score

p_total = p_err + p_format

• Keep cleaner positives (lower p_total) with higher probability.
• Downsample negatives uniformly to preserve failure diversity.

So the algorithm still learns from success/failure, but it is more selective about which successes deserve reinforcement.

rStar2-Agent boosts a pre-trained 14B model to state-of-the-art levels in only 510 RL steps within one week, achieving 80.6% and 69.8% average pass@1 on AIME24 and AIME25, surpassing DeepSeek-R1 (671B) with shorter responses.

Read that again. A 14-billion parameter model, trained for about a week, beating a 671-billion parameter model. And doing it with shorter outputs.

How? Three innovations working together:

1. GRPO-RoC (Resample-on-Correct): A modified reinforcement learning training algorithm that changes how rollouts are selected for gradient updates. Instead of treating all rollouts equally, it oversamples correct trajectories, then filters them by quality — keeping only the ones with the fewest tool call errors and the cleanest answer formatting. Negative (incorrect) trajectories are uniformly downsampled to a fixed count.

2. Reliable Code Infrastructure: The model doesn’t just reason in text. It writes Python code, executes it in a sandboxed interpreter, reads the result, and adjusts its reasoning based on what the code returns. This is multi-turn agentic tool use, and building infrastructure that makes this fast and reliable at training scale was a major contribution.

3. A Multi-Stage Training Recipe: Start with non-reasoning SFT (teaching the model to follow tool-use formatting), then run multiple RL stages with increasing problem difficulty and decreasing max response length. Force the model to get more efficient, not just more accurate.

The paper’s central hypothesis is this: in a noisy tool environment, the quality of the training signal matters more than its quantity. Filtering out the “lucky correct” trajectories that happened to work despite poor tool use is the key move. RoC isn’t about getting more data — it’s about getting cleaner data.

I built a toy notebook to make this intuitive

The paper is rStar2-Agent (arXiv:2508.20722), out of Microsoft Research. The toy is rstar2_agent_toy.ipynb, a single Jupyter notebook using only numpy and matplotlib. No GPUs. No real language model. No Python code interpreter. Just enough scaffolding to understand the algorithm's logic.

Here’s what I learned — and, importantly, where the toy diverges from what the paper actually claims.

I created a runnable notebook that mirrors the paper’s core mechanics with a simplified environment:

• Real LLM → mock policy with controllable skill
• Real math benchmark → synthetic integer math tasks
• Real tool execution → noisy simulator (errors/format issues)
• Full RL weight updates → trajectory sampling + advantage dynamics

🔗 GitHub repo: https://github.com/sulbhajain/agentic-reasoning-toolcalling

Notebook file: rstar2_agent_toy.ipynb

The notebook (rstar2_agent_toy.ipynb) is a deliberately simplified simulation. It demonstrates the structural logic of GRPO vs. GRPO-RoC without any of the actual ML machinery. Here's what it models:

Synthetic math dataset: Problems are generated with known difficulty levels. Each “trajectory” is a simulated multi-step reasoning path — not from a real LM, but sampled with tunable noise parameters.

Tool error simulation: Each rollout has a random chance of triggering a tool error (wrong format, execution failure) drawn from a Bernoulli distribution with rate env_error_rate. This creates the "noisy tool environment" that makes RoC theoretically valuable.

Quality penalty: p_total = p_err + p_format — a combined penalty term that penalizes format violations and tool call errors.

Reward function: Outcome-only reward — 1 if the final answer is correct, 0 otherwise. No step-level rewards.

Vanilla GRPO loop: Standard group relative policy optimization — sample G rollouts per problem, normalize rewards within the group, compute policy gradient.

GRPO-RoC loop: Oversample 2G rollouts per problem, filter the positive rollouts by quality score (keeping the cleanest), downsample negative rollouts uniformly to a fixed count, then compute the policy gradient on this curated batch.

Comparison metrics: Final-step accuracy, response length, tool error rate across training steps — plotted side by side.

https://github.com/sulbhajain/agentic-reasoning-toolcalling/blob/main/rstar2_agent_toy.ipynb

What the toy implementation shows

The notebook includes:

• End-to-end rollout generation
• Vanilla GRPO baseline
• GRPO-RoC training loop
• Comparative metrics: Accuracy, Tool error rate in positive trajectories, Response length
• Visual diagnostics (rstar2_results.png)
• A run-based summary cell that reports observed outcomes without overclaiming replication

The notebook does something genuinely useful: it makes the structure of RoC legible. Before you can understand why RoC might matter at scale, you need to understand the mechanical difference between it and vanilla GRPO.

Running the notebook, you can clearly see:

• The oversample-then-filter loop in action
• How the quality score gates which positives enter the training batch
• That the training loop does produce shorter trajectories under RoC (the notebook reproduces this finding)
• The tension between outcome-only rewards and quality-adjusted sampling

The two-cell walkthrough of a single trajectory trace is particularly good pedagogy — it shows step by step how a rollout is evaluated, scored, and either kept or discarded under each algorithm. If you’ve read the paper and want to develop intuition for the mechanism, running this notebook is 30 minutes well spent.

An important takeaway from building this: run-level behavior in toy environments is highly sensitive to dynamics and hyperparameters. That’s why the notebook now explicitly frames results as run-specific simulation outcomes, not benchmark claims.

Toy implementation diverges

Here’s where it gets interesting.

“GRPO had higher final-step accuracy than GRPO-RoC in the latest run captured in the notebook output.”

That’s the opposite of what the paper claims. And it’s not a bug — it’s a lesson. Let me explain why it happens.

• Gap 1: Random noise vs. structured failure: The paper’s RoC is valuable because tool errors in a real code interpreter are informationally structured. When a 14B model generates syntactically broken Python, that’s not random — it correlates with confused reasoning, wrong intermediate conclusions, and likely wrong final answers. Filtering these out of the positive training set removes trajectories that “got lucky” despite poor reasoning quality. In the toy, tool errors are drawn from env_error_rate as i.i.d. Bernoulli flips. The errors are structurally independent of reasoning quality. RoC's quality filter then has nothing meaningful to select on — it's culling random noise, not correlated failure modes. The asymmetric sampling reduces your effective batch diversity without improving signal quality, which is why vanilla GRPO can win. The paper’s hypothesis only holds when errors are causally linked to reasoning quality. That’s an emergent property of real LM behavior in a real code environment. It cannot be simulated with random noise.
• Gap 2: No policy that actually learns: In the paper, the policy — Qwen3–14B-Base — generates rollouts and then its weights are updated based on the reward signal. Over 510 steps, the model learns to avoid tool errors because cleaner tool use gets rewarded through RoC’s selection mechanism. The compounding effect is critical: as the model improves, the gap between RoC-selected positives and average positives widens, making the selection increasingly meaningful. The toy has no policy that learns. The trajectory generator is fixed — it keeps producing rollouts from the same distribution regardless of what the “training loop” computes. Without a real adaptive policy, you lose the compounding feedback that is the entire point of RL training.
• Gap 3: The penalty term conflicts with the paper’s design philosophy: This is subtle but important. The paper explicitly argues that direct error penalties during early training cause reward hacking and hurt exploration. The answer-only outcome reward (plus RoC’s implicit quality filtering) is the design choice that avoids this trap. The toy applies p_total = p_err + p_format as an explicit penalty — on top of RoC sampling. This conflates two mechanisms the paper deliberately separates. The penalty signal may dominate the RoC filtering effect, which could explain some of the divergence in results. You're testing a hybrid that the paper's design specifically argues against.
• Gap 4: Hyperparameters calibrated to nothing: The official rStar2-Agent training config uses specific, empirically tuned values:

down_sample_to_n=16
min_zero_reward_trace_num=2
min_non_zero_reward_trace_num=2
roc_error_ratio=True
roc_answer_format=True

These numbers were calibrated against real rollout distributions from a 14B language model trained on competition math. The toy’s analogous parameters — oversample ratio, downsample target, penalty weighting — are set without any such calibration. In a synthetic environment with i.i.d. noise, the “correct” hyperparameters are entirely different. Small changes flip which algorithm wins.

Why this matters beyond one paper

This pattern generalizes to a lot of agent systems:

• Browser agents
• Code agents
• Retrieval-augmented planning loops
• Multi-tool orchestrators

Whenever environment feedback is noisy, outcome-only reward can hide poor intermediate behavior. Data/trajectory curation at training time can be a cleaner fix than piling complexity into reward shaping.

In short: keep rewards simple, improve what you reinforce.

If you want to extend this

A few high-impact next experiments:

1. Increase number of seeds and report mean ± std.
2. Compare area-under-curve, not only final step metrics.
3. Sweep environment noise rate to find where RoC helps most.
4. Replace the mock policy with a tiny trainable model.
5. Connect selected trajectories to an actual PPO/GRPO gradient update.

Final thought

Agentic RL’s biggest bottleneck is often not just model intelligence — it’s credit assignment under noisy interaction traces.

rStar2-Agent’s core insight — that signal quality beats signal quantity — is a thesis about what makes reinforcement learning for reasoning work. The paper is arguing that the problem with scaling RL for LLMs isn’t compute or data volume; it’s gradient pollution from low-quality trajectories that happen to produce correct answers through bad reasoning.

This is a much more interesting claim than “we tried harder.” It’s a claim about the mechanism by which RL improves reasoning, and it has implications for how we should think about training reward models, filtering training data, and designing evaluation benchmarks.

rStar2-Agent’s contribution is elegant because it respects that reality: don’t ask rewards to do everything. Use better rollout selection so your model learns from the right signals.

If you’re building tool-using agents, this is one idea worth stealing early.

🔗 GitHub repo: https://github.com/sulbhajain/agentic-reasoning-toolcalling

🔗 Paper: rStar2-Agent: Agentic Reasoning Technical Report https://arxiv.org/abs/2508.20722