Can NeurIPS 25 oral RLVR really improve reasoning ability?

Sharing a NeurIPS 2025 Oral Paper. A Collaboration between Tsinghua University and Shanghai Jiao Tong University

Recently, there has been a lot of buzz surrounding RLVR (Reinforcement Learning with Verifiable Rewards) — the mechanism behind OpenAI-o1 and DeepSeek-R1. The prevailing belief is that this paradigm allows models to acquire new reasoning capabilities through self-play.

However, this paper “throws a bucket of cold water” on that notion. It concludes that current RLVR does not actually make models smarter; rather, it simply makes them better at surfacing capabilities they already possessed.

1. Efficiency Gains vs. Capability Boundaries

The authors utilized a critical metric called pass@k, which measures a model’s performance on a benchmark by giving it k attempts; if it gets the answer right at least once, it is considered a success.

• When k is small (e.g., a single attempt): The RL-trained model is significantly stronger and more accurate. This indicates that RL greatly improves sampling efficiency, making it easier for the model to “stumble upon” the correct path.
• The Turning Point: When k becomes large (providing enough attempts, such as k=256), the performance of the non-RL base model actually surpasses that of the RL model (see Figure above).

Conclusion: This implies that RL does not expand the boundary of a model’s problem-solving ability; instead, it may actually narrow it. Many problems that a base model (or SFT model) could eventually solve given enough tries become unsolvable for the RL model.

2. The “Aha Moment” Was Already There

The “Self-reflection” and “Aha Moments” highlighted in DeepSeek-R1 look like brand-new capabilities. However, by analyzing perplexity metrics, this paper found that the reasoning paths generated by RL models were actually already present within the distribution of the Base Model.

Therefore, RLVR isn’t teaching the model new knowledge or new logic. Instead, it is acting as a filter within the Base Model’s vast potential solution space — selecting correct paths and suppressing incorrect ones. Consequently, the Base Model serves as the “ceiling” for the RL model.

3. Distillation vs. RL

The authors also drew an interesting comparison with Distillation. They found that distillation can truly teach a student (a weaker model) new reasoning patterns from a teacher (a stronger model), thereby expanding the student’s capability boundary.

This is fundamentally different from the logic of RLVR, which merely mines the model’s own internal potential. The paper argues that current RLVR lacks an effective Exploration mechanism. In a massive Action Space like natural language, it is difficult for models to discover entirely new strategies through simple reward patterns; they end up just “circling around” the prior knowledge learned during pre-training.

| Find papers faster on arXivSub with AI summary (CVPR/ICCV/ICML/ICLR/NeurIPS/AAAI/MICCAI)