From RLVR to Curriculum RFT and back: Building a Self-Improving GSM8K Pipeline for Qwen

In the last post, I implemented Reinforcement Learning with Verifiable Rewards (RLVR) from scratch and trained Qwen-2.5–0.5B on GSM8K using the REINFORCE algorithm.

While REINFORCE did improve the model, I wanted to answer a more fundamental question :

What is the actual reasoning potential of this model?

Can the model already generate correct solutions but simply fail to select them consistently? Or is the reasoning capability itself missing? To answer this, I explored Best-of-N evaluation, rejection sampling, curriculum-based Rejection Fine-Tuning (RFT), and finally did GRPO on the final model.

This post talks about the entire pipeline and also references the code and the lessons learned while trying to maximize pass@1 accuracy.

GRPO vs REINFORCE: Moving from sequence-level to more stable token-level optimization

Before doing any rejection sampling or curriculum learning, I wanted to understand how much improvement could be obtained simply with a stronger policy optimization algorithm.

In the REINFORCE setup from Part 1, the model also receives the same group-normalized advantage signal for the entire generated sequence like in GRPO. But, I summed the token log-probabilities over the completion to get a sequence-level log-probability, which treats the sampled completion as one trajectory. In the GRPO version, I used the same advantage, but applied it through a PPO-style token-level objective. Instead of directly optimizing the summed sequence log-probability, GRPO first, computes the ratio between the current policy and the old policy at each token:

Fig. The current ratio (the current model being trained), the old ratio (the model used to sample rollouts).

and then also apply clipping :

Fig. Clipping for maintaing training stability.

At the end, I also added a KL penalty against the reference model to prevent the policy from drifting too far. This made GRPO more stable.

I trained the same Qwen2.5–0.5B model with GRPO for 1500 steps and compared it against the REINFORCE model.

Note : My REINFORCE baseline is stronger than vanilla single-sample REINFORCE. I used the same group-normalized advantage computation for both REINFORCE and GRPO i.e; for each prompt, I sampled multiple completions, scored them with the verifier, and normalized the rewards within that group. This gives REINFORCE some of the same variance-reduction benefit that is usually associated with grouped sampling. So the comparison is conservative for GRPO: the improvement does not come from a different reward or advantage computation, but from how the policy loss applies that signal.

Fig. Extending the graph from Part I, the GRPO model improved the performance on pass@1 at 1500 steps, but falls abit short with REINFORCE on pass@3 and pass@5 accuracy.

GRPO gives a further 3.59% improvement in Pass@1 but remains shy in performance for Pass@3 or Pass@5. This is a known challenge in RL for reasoning models [1, 2]. One possible interpretation is that GRPO shifts more probability mass toward a smaller set of high-confidence responses, improving first-sample reliability while slightly reducing the benefit of sampling multiple completions. This means your top 3 or top 5 answers may look slightly identical, while reducing diversity among your top outputs.

🙋 — Why GRPO is better suited than REINFORCE for this setup ?

Consider the following GSM8K-style reasoning problem:

Kylie has 34 stamps in her collection. Her friend, Nelly, has 44 more stamps than Kylie. How many stamps do Kylie and Nelly have together?

The correct reasoning is straightforward:

Nelly = 34 + 44 = 78 Total = 34 + 78 = 112 stamps

The GRPO-trained model follows this directly:

Kylie has 34 stamps. Nelly has 44 more stamps than Kylie, so Nelly has (34 + 44 = 78) stamps. Together they have (34 + 78 = 112) stamps. Therefore, the answer is (112).

The interesting part is one of the REINFORCE samples. The model sets up the problem correctly:

Nelly’s stamps = Kylie’s stamps + 44 Total stamps = Kylie’s stamps + Nelly’s stamps

It even writes the correct Python-style calculation:

kylie_stamps = 34
nelly_stamps = kylie_stamps + 44
total_stamps = kylie_stamps + nelly_stamps
print(total_stamps)

But then the final answer becomes: \boxed{108}. The model did the hard part correctly, then somehow tripped over the calculator on the way out.

This is a good example of why sequence-level optimization can be noisy for reasoning problems. With REINFORCE, the whole sequence is pushed down, even though many of its tokens correspond to good reasoning.

But with GRPO, its loss applies the same advantage in a more controlled way: token-level policy ratios, clipping, and KL regularization prevent the policy from making overly large updates from a single sampled trajectory.

✅ — This is why GRPO is better suited for this setup. The reward is sparse and sequence-level, but the optimization does not have to be as blunt as sequence-level REINFORCE.

🙋 — How much reasoning ability is already inside the model’s sampling distribution?

Based on the results above, I learned that the model reached 60.89% Pass@3 and 68.01% Pass@5, while Pass@1 was only 44.17%. At this point, Pass@1 was telling me how reliable the model was, but Pass@k was telling me how much potential was still hiding in the model’s sampling distribution. That gap became the motivation for Best-of-N.

Measuring the Upper Bound with Best-of-N :

After seeing that sampling variance mattered, I wanted to estimate the model’s maximum potential. Pass@1 only answers one question:

Does the model’s first sampled answer solve the problem?

But for a small reasoning model, this can underestimate capability. A model may fail on its first attempt but still generate a correct solution if sampled multiple times.

So I evaluated the model with Best-of-N on GSM8K and AIME, N ∈ { greedy, 3, 5, 8, 16}. The idea is simple:

1. For each problem, sample N candidate solutions.
2. Use a verifier to check whether each candidate reaches the correct final answer.
3. Count a problem as solved if at least one of the N samples is correct.

This gives a rough estimate of the model’s hidden reasoning potential.

Fig. As the number of sampled completions increased, performance improved sharply for qwen2.5–0.5B model.

It shows that the model can solve many more GSM8K problems than its first sampled answer suggests. The correct solution is often present somewhere in the model’s output distribution, but it is not always the first trajectory sampled.

I also ran the same evaluation on AIME. The results were much weaker.

Greedy accuracy was 0%, and even with multiple samples the model barely solved any problems. At Pass@5, the model reached only 2.33%, and at Pass@16 it was still 0% in this run. This was an important contrast with GSM8K.

On GSM8K, Best-of-N revealed a large hidden reservoir of correct reasoning. On AIME, that hidden potential was mostly absent for this model at this stage. The model was not merely failing to select the right solution; it generally could not generate correct AIME solutions reliably.

If the model cannot solve a problem in any sampled rollout, then there is no useful self-generated solution to train on. But once I saw that correct answers existed somewhere in the rollout distribution, the next step was to stop treating those rollouts only as artifacts and rather as training data.

This leads naturally to rejection sampling.

Rejection Sampling :

Since the model could frequently solve problems under Best-of-N sampling, I used rejection sampling to convert those successful rollouts into high-quality supervised training data. For each GSM8K training problem, I generated multiple rollouts from the model and measured each problem's difficulty. Each rollout was scored using the verifiable reward function and ranked each correct sample based on completion length. If the final answer matched the ground truth, the sample was kept. If not, it was rejected.

This created a dataset of model-generated correct solutions.

Instead of training on every generated sample, I only trained on samples that passed the verifier. This allowed the model to learn from its own successful reasoning traces.

Building two RFT datasets : top-2 vs all-correct from GSM8K train set

Top-2 correct samples : For each problem, I selected up to top-2 correct completions based on correct samples ranking.

The motivation was to keep dataset cleaner and more balanced per problem, but it may throw away useful diversity.

All-correct samples : For each problem, I kept all correct completions generated during rejection sampling.

The motivation was to preserve the full distribution of successful reasoning traces.

Fig. Problem Difficulty level for GSM8K train set. For very hard problems, none of the generated rollouts solved the problem, so the only available trajectory was the original GSM8K training solution. Success ratio is defined by how many successful rollouts were able to generate the correct response.

I did not train only on rejected-sampled model outputs. I combined the generated RFT datasets with the original GSM8K training set.

This helped preserve the original supervised signal while adding model-specific successful reasoning traces.

Curriculum based RFT :

As mentioned earlier with vanilla Rejection Fine Tuning (RFT), easy problems may naturally contribute more samples while harder problems contribute fewer samples because of the sample dominance in the dataset.

The model first trained on problems it could solve more reliably, then gradually moved toward harder problems.

This is similar to how humans learn: first consolidate what is already within reach, then expand into harder examples.

Fig. Training progress with curriculum learning. Initially model is fed with more easy samples and then harders problems are being introduced gradually. After a certain point in training step(s), a single batch contains a fixed percentage of different difficulty problems.

For this model and dataset, keeping all verified correct trajectories (all-correct dataset) gave a stronger training signal than aggressively limiting the number of completions per problem. [ Not mentioning results due to length — but try to think why so ].

Frontier labs may very carefully craft the distribution of different samples. But for this toy project, I only did 3 phases as explained in the figure above.

Fig. Comparison on RFT model with pre-RFT original vs pre-RFT GRPO / REINFORCE model(s).

It is a modest regression for RFT model to under-perform pre-RFT GRPO.

This was a useful reminder that “correct” does not always mean “optimal for training.” RFT optimizes likelihood, not reward. Even if every generated completion is verifier-approved, SFT still treats verbose, brittle, or oddly formatted correct solutions as valid targets. GRPO, on the other hand, continues to optimize against the verifier reward directly. Readers can also try marking problem difficulty based on the completion token(s) length.

This suggested that RFT was useful as a way to reshape the model’s distribution, but not necessarily as the final optimization step. The final model still needed another round of GRPO on top of the RFT checkpoint to convert the filtered reasoning traces back into reward-optimized behaviour.

Fig. Recap of the blog. GRPO showed that better policy optimization improved first-sample reliability. Best-of-N showed that the model had more reasoning potential than Pass@1 suggested. Rejection sampling extracted the successful rollouts. Curriculum RFT tried to distill them back into the model.

So the lesson from this stage was simple: RFT helped reshape the model, but it did not replace reward optimization. The next step was to take the best RFT checkpoint and put it back into GRPO.

But that is for the next blog, because this one has already asked Qwen to do enough math for one day.