Implementing RLVR from Scratch: Training Qwen2.5 with REINFORCE on GSM8K

Building a minimal RLVR pipeline from scratch, understanding sequence log-probabilities, reward normalization, and training a reasoning model using exact-match rewards.

Motivation :

Andrej Karpathy recently (well, not recently in terms of LLM research advancement) talked in Dwarkesh Patel's podcast and mentioned :

Otherwise you are missing on the knowledge.

So I wanted to build something simple and to show how easy it is to implement Reinforcement Learning with Verifiable Rewards (RLVR) and get it working. I decided to keep the setup intentionally simple. Instead of relying on existing frameworks such as TRL or OpenRLHF, I built a minimal implementation from scratch. The goal here is not to make another state-of-the-art model but rather to understand :

• Rollout generations
• Reward computations
• Advantage estimations
• Sequence log-probabilities
• Policy gradient updates

from a code perspective. The code is here : https://github.com/sanketsans/rlvr. Wherever relevant, I have linked the individual RLVR components discussed in this blog directly to their implementation in the codebase.

Understanding RLVR concepts :

There are great resources online for learning about RLHF / RLVR. Here are some of my recommendations :

1. Books : https://rlhfbook.com/ , https://web.stanford.edu/class/psych209/Readings/SuttonBartoIPRLBook2ndEd.pdf
2. Youtube : https://www.youtube.com/watch?v=PAz_-xPJcRM&t=615s , https://www.youtube.com/watch?v=o6l6tJQgUg4&t=921s

The basic idea is explained in the figure below ( generated using flipbook )

Fig. 1. — For a given prompt — generate many rollouts. 2. — Sample model's response (make sure temperature is set high for different response). 3. — Compute rewards & advantages for the generation using simple verifiable method (exact match or compilers for code execution). 4. — Update the policy based on those rewards.

🙋 — Why we need multiple generations i.e; rollouts ?

✅ —RLHF and RLVR are particularly effective when there is variation in the quality of the generated responses. As we will see later, when every response receives the same reward, the learning signal can disappear entirely.

Ok, Lets talk code now.

I divided the implementation in two phases :

Phase 0 : Eval implementation. I measure pass@k to better evaluate the sampling efficency of the model.

Phase 1 : Training using RLVR implementation.

This post will mostly cover Phase 1 — because Phase 0 is very straightforward. I compute Pass@1, Pass@3, and Pass@5 on the GSM8K test set to evaluate model performance.

Rollout Generations :

I set the sampling temperature of the model to 0.7 to generate diverse completions from the model for a given prompt. I mention in the system prompt to generate answer precceded by ####(that's the format used by GSM8K for their samples).

Verifier :

I extract the answers from GSM8K samples and the completions and simply check the exact match to compute my binary reward (0 / 1). So for a single prompt we can have rewards like :

Question: What is 12 × 25?

Completion 1: 300 -> Reward = 1 Completion 2: 250 -> Reward = 0 Completion 3: 300 -> Reward = 1 Completion 4: 320 -> Reward = 0

Instead of treating these rewards independently, the policy-gradient update compares completions within the same rollout group. The intuition is simple:

• Completions that perform better than the group average should become more likely.
• Completions that perform worse than the group average should become less likely.

This relative comparison is what advantage estimation captures.

Advantage estimations :

Advantage estimation, measures how much better or worse a completion is relative to other completions generated for the same question and can be done using this simple formula :

advantages = (rewards — rewards.mean()) / rewards.std()

The policy gradient algorithm then increases the probability of positive-advantage completions and decreases the probability of negative-advantage completions.

Important observations while debugging : Initially, I kept getting mean_advantage=0.0 or advantage= 0.0 (for all completions).

• Mean advantage = 0.0 — For the example mentioned above with rewards:

[1, 0, 1, 0] -> advantage = [+0.87, -0.87, +0.87, -0.87]

So, mean_advantage eventually cancels out positive and negative values. However, seeing an advantage mean close to zero is actually a useful sanity check that the normalization is working correctly.

• Advantage = 0.0 — During initial / later phase(s) of training, you might encounter instances / batches where :

Advantage = [0.0, 0.0, 0.0, 0.0]

These cases may arise from batches where the models got everything correct / wrong.

Rewards = [1, 1, 1, 1] or [0, 0, 0, 0]

We might be tempted to see that the model be improving. However, these batches provide no learning signal because every completion receives the same reward. Since advantage-based methods only know that every completion performed equally poorly. They do not know which completion should be preferred. To reduce the frequency of these cases :

• Increase the max_new_tokens limit — If the model frequently truncates before producing a final answer, every completion may receive a reward of 0, eliminating useful learning signals. One might argue though, that we can train the model to then generate response with lower token limit — But that may take a loooong time.
• Increase batch size / rollouts/ temperature — I played around with the batch size and the number of rollouts and the temperature of the model to generate diverse responses for each question(s).

REINFROCE objective :

The next challenge was computing the quantity that actually appears in the REINFORCE / GRPO objective:

Fig. Loss function for REINFORCE algorithm.

where:

• x is the prompt
• y is the generated completion
• A is the advantage
• log π(y|x) is the log-probability of the generated completion under the current policy.

🙋 — Why not simply ask the model for the probability of the completion?

✅ — The challenge is that language models are autoregressive. They do not assign a probability to the entire completion in a single step. Instead, they predict one token at a time, conditioning on all previous tokens.

Lets take the example mentioned above for the prompt and the model generates a completion i.e; 300. The model sees it as :

P1 P2 P3 C1 C2–3 prompt tokens, 2 completion tokens) \

During training (and when computing log-probabilities), the entire sequence is fed into the model at once. Thanks to causal masking, each position can only attend to previous tokens, meaning the model learns to predict the next token. We use the same idea here to compute the log-probs only on the completion tokens — cuz that's what matters. So,

• Extract the logits from the model for the entire sequence : prompt + completion. — During batching, padding tokens may also be present, but they are masked out later and do not contribute to the final sequence log-probability.
• Shift the logits and targets by one position so that every prediction is aligned with the token it is supposed to predict.
• Convert logits to log probabilities using softmax.
• Extract the log-probability assigned to the token that actually occurred at each position.
• Filter using attention mask and prompt length, to extract and sum the token log-probs across all the completion tokens.

Fig. sequence log probabilities

This quantity (sequence log probabilties) is one of the fundamental building blocks behind modern post-training algorithms including REINFORCE, PPO, GRPO. While the optimization objectives differ, they all rely on measuring how likely the model considers a generated completion under its current policy.

Experimental Setup and Results :

Model : I used the Qwen2.5–0.5B-Instruct model because it is small enough to iterate on quickly and has not undergone the extensive reasoning-focused post-training used in newer models such as Qwen3.

Dataset : I used GSM8K dataset for training and evaluation — it consists of grade-school mathematical word problems requiring multi-step reasoning and arithmetic. eg.

“Every day, Wendi feeds each of her chickens three cups of mixed chicken feed, containing seeds, mealworms and vegetables to help keep them healthy. She gives the chickens their feed in three separate meals. In the morning, she gives her flock of chickens 15 cups of feed. In the afternoon, she gives her chickens another 25 cups of feed. How many cups of feed does she need to give her chickens in the final meal of the day if the size of Wendi’s flock is 20 chickens?”

While GSM8K is largely saturated by frontier reasoning models today, it remains one of the most widely used benchmarks for studying reasoning capabilities and post-training methods. But it works fine for our case to formulate easy reward computation and not need training a separate verifier model. My reward computation :

reward = 1 ; if answer exactly matches the answer

reward = 0 ; otherwise

I ran multiple experiements with different learning rate (1e-5, 1e-6, 5e–7) for different step update(s) with different batch size. I am not going to discuss all the results — but important ones. I realized that a lower learning rate help the model steer to better predictions over time :

Fig. lr=5e-7 with batch size = 16

compare to a higher learning rate which overfits quickly :

FIg. lr=1e-5 with batch size = 16

Note : During training, I evaluated on a subset of 200 GSM8K test examples to reduce evaluation overhead. The final results reported below are computed on the full GSM8K test set.

It is imperative to keep logging your average rewards, advantage over the duration to make sure that the model is getting effective learning signals over time. Throughout training, I monitored the reward and advantage standard deviations to ensure that the model continued to receive meaningful learning signals.

Fig. rewards & advantage standard deviation over the training step(s).

Also, I believe the model still has capacity to learn more just by looking at the gradual decrease in my loss curve :

Fig. Loss curve with lr=5e-7 and batch size=16. Make look static at first glance but keep looking :)

After 200 RLVR training steps, the model consistently improved across all GSM8K pass@k metrics.

The largest absolute gain was observed in Pass@5, which improved from 64.97% to 70.05%, while Pass@1 improved from 36.66% to 40.58%. Although the training setup was intentionally simple — using exact-match rewards and the vanilla REINFORCE algorithm — the model was still able to learn a stronger reasoning policy from verifier feedback alone.

Fig. RLVR training with REINFORCE for 200 steps improves the Qwen2.5–0.5B model across all metrics. While Qwen3–5B (post-trained) performs substantially better, the consistent gains indicate the effectiveness of RLVR and the potential for further improvements with stronger algorithms like GRPO.

An interesting observation is that the gap between the RLVR-trained model and the larger Qwen3–5B model remains substantial, particularly for Pass@3 and Pass@5. This is expected for several reasons:

• Qwen3–5B is significantly larger.
• It has undergone extensive supervised fine-tuning and post-training.
• Modern post-training pipelines typically use more sophisticated optimization methods such as PPO or GRPO rather than vanilla REINFORCE.

However, the results provide encouraging evidence that even a minimal RLVR implementation can produce measurable improvements in mathematical reasoning performance.

More importantly, this experiment validates the end-to-end RLVR pipeline:

1. Generate multiple rollouts per question.
2. Compute verifier-based rewards.
3. Estimate advantages.
4. Optimize sequence log-probabilities using policy gradients.

Pro Tip : Batch everything. Moving from per-sample computation to batched rollout generation, reward computation, and sequence log-probability evaluation dramatically reduced iteration time and made experimentation much easier.

The goal of this project was never to train the best GSM8K model, instead to understand RLVR from first principles and build every component myself. Seeing measurable improvements from a few hundred steps of REINFORCE training was enough to convince me that the pipeline works — and that there is plenty of room to explore stronger methods such as GRPO next.