Overview of Reinforcement Learning Methods for Enhancing VLA Robot Training

The Vision Language Action (VLA) Models have gained tremendous attention in the past couple of years, thanks to the great progress made in the field of large language model (LLM) and vision language model (VLM). Among the very first VLA models are OpenVLA [1] proposed by Stanford researchers. The generative model used by OpenVLA is a diffusion based model that is slightly unstable and also slow during inference. Physical Intelligence addressed these limitations with π0 [2] and π0.5 [3] models, which combine VLM backbones with flow matching for continuous action generation — which has shown amazing performance if the model went through a phase of supervised fine-tuning on the robot trajectory data in the environment under deployment.

The problem formulation of SFT loss is shown below where the network tries to minimize the distance between ground truth action chunks of length H and the generated actions chunks from the diffusion policy for K_i observed trajectories:

If you are interested you can check my previous blog on π0 and π0.5 on medium.com here:

[embed]Robotics Foundation Models by Physical Intelligence: OpenPi Introductionmedium.com

The SFT seems a high burden in new environments and without that the robot seems to fail miserably when doing new tasks on new objects and in new environments that was not used during fine-tune step. The robot somehow only memorized the actions during SFT and doesn’t learn them as preferred when tested in new environments and on new objects! Therefore, there has been some efforts to enhance the performance of VLA models using reinforcement learning (RL) which is the traditional approach to robot training. In the following I describe a few recent papers and research works in this domain [4–6].

RL-based VLA Training

1- RL-VLA :

One of the recent works in this domain, which is published in Neurips 2025 [4], is a PPO-based model applied to OpenVLA foundation model. It uses an actor critic architecture where the Value is predicted using an MLP network applied on the last transformer block of OpenVLA before generating action tokens [4].

The PPO is among the policy gradient methods composed of two neural networks; actor and critic. The actor network predicts the actions based on the recent policy. Then the critic tries to predict the state values. A reward is assigned to the actions taken and then an advantage value is calculated which shows how much the return is better than state value estimated by critic network. During train the critic tries to minimize the loss between true return values and it’s prediction. The critic is not independent than actor as it’s prediction of value affects the advantage.

In RL-VLA [4], The actor is the openVLA robot policy that is being fine-tuned using LoRA. To save VRAM and speed up training, the actor (policy) and critic (value function) share the same Transformer backbone. In summary the RL-VLA algorithm is trained as follows:

The robot uses its current policy to rollout an action and the log probability, entropy and rewards are returned.

The relative return and generalized advantage estimation (GAE) which is the relative improvement of an action compared to average of other actions are estimated through the trajectory for each time-step t as:

for t in reversed(0 to T-1):
    # TD error
    δ_t = r_t + γ × V(s_{t+1}) × mask_{t+1} - V(s_t)

    # GAE accumulation
    GAE_t = δ_t + γ × λ × mask_{t+1} × GAE_{t+1}

    # Returns (for value function target)
    Returns_t = GAE_t + V(s_t)

Advantages = Returns - V(s_T) 

# Normalize advantages
Advantages_normalized = (Advantage - mean(Advantages)) / (std(Advantages) + 1e-5)

The advantage is then calculated as the difference between the returns and the last value estimate. The advantage is an indicator for policy; if positive the action is considered a good move, else if negative the action was bad and will be penalized. Based on the normalized estimated advantage of an action, A, and the log probability, the surrogate policy loss is calculated.

Loss_policy= -min(r·A, clip(r)·A)

where r = π_new(a|s) / π_old(a|s) is the ratio of new policy to old policy and clip keeps the parameter within [1-ε, 1+ε]. The negative of loss is minimized which is equivalent to maximizing the reward policy of actions, as the actor tries to maximize the policy and take actions that have high advantage.

The value is predicted by critic network (MLP) from the hidden layer output of openVLA right before action tokens. The critic tries to minimize the distance between the estimated return coming from the MLP head and the return calculated from rewards. The value loss is calculated using a clipped huber loss, which is a robust L1-L2 trade off as

L_value: Huber( V_clipped — Returns_t)

where V_clipped = V_old + clamp(V — V_old, -ε, ε) and clamp is clipping the value between [-ε, ε].

Finally, the loss is formed as the sum of policy, value and an optional entropy loss for exploration. The architecture of the RL-VLA is shown below.

RL-VLA model architecture applied to OpenVLA

The findings show that the RL-based training can gain significantly on execution of tasks compared to SFT. RL-trained models can recover from mid-episode object repositions and handle unseen robot initial poses. While SFT agents often “march on” despite missing a grasp, RL agents learned to retry and adjust their end-effector poses. On semantic grounding where robot is supposed to generalize to unseen objects and novel instruction phrasing, there was moderate gains. On the vision side the RL didn’t provide much advantage over SFT.

2-π_RL [6]

This work is from some of the authors of RL_VLA so it is an improvement by exploiting π0.5​ instead of OpenVLA as SOTA and applying RL. Since flow matching output tokens are deterministic and not suited for RL exploration during rollout, the framework introduces two solutions to characterize the logarithmic likelihood of executed actions:

• Flow-Noise: This approach integrates a learnable noise network into the denoising process. By modeling the denoising stage as a discrete-time Markov Decision Process (MDP), the system can directly compute the exact log-likelihood of the de-noised action sequence for policy optimization.
• Flow-SDE: This method is based on the idea of flow-GRPO that uses RL for enhancement of flow matching, and it converts the deterministic Ordinary Differential Equation (ODE) process of flow matching into a Stochastic Differential Equation (SDE). It creates a two-layer MDP that couples the internal denoising steps with the external agent-environment interaction, allowing for efficient exploration while maintaining equivalent marginal distributions.

The π_RL has these steps:

• Selective Fine-tuning: To ensure GPU memory efficiency, the large VLM backbone is frozen, and RL fine-tuning is applied exclusively to the action expert.
• Critic Design: In the π0.5​ variant, the critic network (responsible for value prediction) is attached directly to the VLM output. This is because π0.5​ merges proprioceptive state information with prompt embeddings within the VLM, rather than feeding it separately into the action expert.
• Policy Optimization: The framework uses the Proximal Policy Optimization (PPO) algorithm to optimize the policy parameters. For π0.5​, a learning rate scheduler with cosine annealing is often employed to stabilize training and prevent the escalation of KL divergence.

The GAE is estimated similar to RL-VLA.

Experimental results show that applying πRL​ to π0.5​ yields substantial improvements over supervised fine-tuning (SFT) baselines:

• Success Rates: In the LIBERO benchmark, πRL​ achieved a 98.3% success rate, surpassing the full-dataset SFT baseline of 96.9%.
• Generalization: The framework effectively enhances the model’s ability to handle out-of-distribution (OOD) visual and execution variations, suggesting the acquisition of generalized action representations rather than narrow overfitting.
• Efficiency: It significantly improves temporal efficiency, with episode lengths converging to expert-level ranges after RL training.

3- π∗0.6:

The Physical Intelligence group introduced π0 and π0.5 and later an enhancement called π0.6 [4]. An RL-based enhancement to it called π0.6 is released also named RECAP (Reinforcement learning with Experience and Corrections via Advantage-Conditioned Policies), a general-purpose recipe for RL training of large-scale VLAs. The idea of π∗0.6 is based on π*0.6 by conditioning the model on a binarized advantage indicator (I_t), which tells the model if the action the robot is taking is optimal or not, in the sense of task completion.

Step I: data collection with human correction:

The human in the loop starts introducing randomness in the scene by moving objects while the robot is doing actions. This helps increase robustness of the robot to failures and learning to take steps to get rewarded.

Step II: Value function training:

To act as a reliable critic, π∗0.6 uses a distributional value function pϕ​(V∣o_t​,ℓ) that maps observations and language commands to a distribution over B discretized value bins. Let R_t(τ) be the empirical return of a trajectory τ from step t until T. The empirical return is discretized into B=201 bins, and then cross-entropy loss H is minimized over empirical returns R_t​(τ) as

The value function is then estimated from the learned value distribution

where v(b) is the value for bin b. This value function identifies mistakes on-the-fly, allowing the system to estimate the “advantage” of any given action — essentially asking, “Is this action better than what robot usually does?”

Step III: Advantage conditioning training:

Once the value function is available, it should be used to train an improved policy, which is known as policy extraction method. While there are some well-known RL methods, the authors claim that those are apparently hard to be applied to flow matching models. The authors proposed an advantage conditioning where an additional input indicating how optimal the action might be, is introduced as indicator:

where the indicator I(.) is modeled as a delta distribution where it is 1 when the advantage is greater than threshold ϵ_ℓ and 0 otherwise. Therefore, the policy objective is to minimize

The advantage value A(o_t, a_t, l) are obtained from the value function calculated earlier.

Step IV : Continuous control with flow matching

Because π∗0.6 must generate smooth, continuous actions for dexterous tasks like tamping espresso or folding laundry, it integrates the flow-matching loss into its RL objective. The model optimizes a lower bound on the action likelihood:

Here, f_θ​ represents the continuous output of the action expert, and ω is sampled noise. This formulation allows the VLA to maintain the benefits of expressive action distributions while benefiting from RL-based improvement.

The π06 diagram of training architecture.*

By deploying π∗0.6 in a continuous loop — collecting autonomous rollouts, receiving human corrections, and retraining — the model achieves performance levels far beyond standard supervised learning in terms of throughput gain, failure reduction and robustness for long period of time.

In Conclusion, π∗0.6 demonstrates that RL for VLAs doesn’t have to be unstable or complex. By treating RL as a conditioning problem and leveraging the power of flow matching, we can move from generalists that can do a task to experts that do it well.

The only limitation is that the robot needs to be trained with a human supervision for many examples so it can master the tasks.

Conclusion

The use of RL as en enhancement to the VLA is a promising trend in recent years and seems to be a preferred approach as compared to pure SFT. Since applying RL to VLA is not as straightforward as conventional robot learning, the papers discussed have applied novel RL ideas to the VLAs. While these approaches seem to improve the results compared to SFT, still there is room for developing more optimal RL formulation that can generalize to unseen environments and tasks.

References

1- https://arxiv.org/abs/2406.09246

2-https://www.pi.website/blog/pi0

3-https://www.pi.website/blog/pi05

4-What Can RL Bring to VLA Generalization? An Empirical Study

5-https://www.pi.website/download/pistar06.pdf

6-https://arxiv.org/pdf/2510.25889