MM-EUREKA: Is RL needs Small amount of data, multiple episodes?

Introduction

We present MM-Eureka-Qwen, a multimodal reasoning model that successfully extends large-scale rule-based reinforcement learning (RL) to multimodal reasoning. Compared to the previous version of MM-EUREKA based on InternVL, we have made improvements in model architecture, algorithms, and data. Using only non-in-domain training data, MM-Eureka-Qwen achieves significant improvements over Qwen-2.5-VL-Instruct-7B across multiple benchmarks (e.g. MathVista 73.0). We release all our codes, models, data, etc. at https://github.com/ModalMinds/MM-EUREKA.

Improvements:

1. We further iterate the codebase to support algorithms including Online Filter, ADORA, and DAPO.
2. We expand our K12 dataset, collecting 15,000 high-quality K12 samples.
3. We train the MM-Eureka-Qwen-7B model with GRPO and Online Filter, achieving better results with significantly lower cost than the previous version. We open-sourced our training code and models and hope to facilitate future studies on multimodal reasoning.

MM-EUREKA-Qwen

Based on the key factors identified by https://github.com/ModalMinds/MM-EUREKA for achieving stable training, we enhanced the model, dataset, and algorithmic modules. Specifically, we maintained the strategy of omitting the KL divergence term and applying data filtering, while implementing the following critical modifications:

• The base model was upgraded from InternVL2.5–8B-Instruct to the more powerful QwenVL2.5–7B-Instruct.
• The Vision Transformer (ViT) module was kept fixed during training.
• The underlying RL algorithm was replaced with GRPO, instead of the previously used RLOO.
• The data filtering strategy was transitioned from an offline approach to an online approach.
• Additional data from the K12 dataset was collected, expanding the total dataset size to 15,000 samples.

Finally, MM-EUREKA-Qwen achieves 73.0 on MathVista, surpassing the original Qwen-2.5-VL by 4.8%.

Training Recipe

Compared to previous MM-EUREKA-InternVL, we fix the Online Filter issues. Specifically, we believe that the original reason why the accuracy reward and length could not increase steadily is that the actual batch size used each time the actor is updated after the filter was different. Therefore, to maintain consistent batch sizes after filtering, we keep a pool equal to the target batch size.

For data management in this pool, we introduced two strategies: flush and clear strategy. In both strategies, the policy model is updated only when the number of samples in the pool reaches the target size. However, they differ in handling samples that exceed the target size during online filtering: the flush strategy retains the surplus samples in the pool for subsequent policy model updates, whereas the clear strategy discards them entirely. The detailed process is shown in the figure below.

For the RL training pipeline, we adopt the GRPO algorithm and employ a rule-based reward function which comprises both an accuracy reward and a format reward. In our preliminary experiments, both strategies achieve similar results. In this report, we mainly use the clear strategy.

Experiments

We first describe the details of our model training process. In terms of data, we expand upon the 8,000 K12 samples from the open-source MM-EUREKA Dataset by collecting an additional 7,000 K12 samples. It is important to note that these data completely exclude the corresponding training datasets of our evaluation benchmarks (e.g., Mathvista). Despite this separation, our model demonstrates substantial performance gains on these benchmarks, underscoring its strong generalization capability.

For the specific training strategy, we fine-tune the Qwen2.5VL-7B-Instruct model using an Online Filter strategy using the script below. We use the final checkpoint as our final model, with no early stopping applied.

export RAY_MASTER_PORT=6379
export RAY_DASHBOARD_PORT=8265
export NCCL_TIMEOUT=7200

export DATASET="k12.jsonl"

OUTPUT_DIR='K12_onlinefilter_Episode10'
PRETRAIN_MODEL="Qwen2-5-VL-7B-Instruct"
export REWARD_LOG_PATH="${OUTPUT_DIR}/reward.log"

if [ ! -d "$OUTPUT_DIR" ]; then
  mkdir -p "$OUTPUT_DIR"
fi

if [ "$PET_NODE_RANK" -eq 0 ]; then
    ray start --head  --port=$RAY_MASTER_PORT --dashboard-host=0.0.0.0 --dashboard-port=$RAY_DASHBOARD_PORT --num-gpus 8
else
    sleep 30
    ray start --address="$MASTER_ADDR:$RAY_MASTER_PORT" --num-gpus 8 --block
fi

sleep 30

if [ "$PET_NODE_RANK" -eq 0 ]; then
  cd mmeureka-qwen
  export WORKING_DIR=/mmeureka-qwen
  RAY_ADDRESS="<http://127.0.0.1>:$RAY_DASHBOARD_PORT" ray job submit \\
    --working-dir $WORKING_DIR \\
    --runtime-env mmeureka-qwen/runtime_env.json \\
    -- python3 -m openrlhf.cli.train_ppo_ray \\
    --remote_rm_url examples/scripts/reward_func_qwen_instruct.py \\
    --actor_num_nodes 2 \\
    --actor_num_gpus_per_node 8 \\
    --vllm_num_engines 8 \\
    --vllm_tensor_parallel_size 1 \\
    --vllm_gpu_memory_utilization 0.3 \\
    --pretrain $PRETRAIN_MODEL \\
    --save_path ${OUTPUT_DIR} \\
    --micro_train_batch_size 2 \\
    --train_batch_size 128 \\
    --micro_rollout_batch_size 2 \\
    --rollout_batch_size 128 \\
    --temperature 1.0 \\
    --n_samples_per_prompt 8 \\
    --lambd 1.0 \\
    --gamma 1.0 \\
    --max_epochs 1 \\
    --num_episodes 10 \\
    --prompt_max_len 3000 \\
    --max_samples 100000 \\
    --generate_max_len 4096 \\
    --advantage_estimator group_norm \\
    --zero_stage 1 \\
    --bf16 \\
    --actor_learning_rate 1e-6 \\
    --init_kl_coef 0.0 \\
    --prompt_data $DATASET \\
    --disable_fast_tokenizer \\
    --input_key message \\
    --adam_offload \\
    --flash_attn \\
    --gradient_checkpointing \\
    --save_steps 50 \\
    --ckpt_path "${OUTPUT_DIR}/ckpt" \\
    --max_ckpt_num 1000000 \\
    --save_hf_ckpt \\
    --freeze_prefix visual \\
    --enable_accuracy_filter \\
    --accuracy_lower_bound 0.1 \\
    --accuracy_upper_bound 0.9 \\
    --use_tensorboard "${OUTPUT_DIR}/tensorboard" \\
    --load_checkpoint | tee ${OUTPUT_DIR}/training.log
fi
ray stop

The data should be organized in the following format. Absolute paths are required for image files. For text-only data, the entry {"type": "image", "image": "04cf92d72682f8796387e849b4bb4.png"} can be removed. The training framework supports mixed-modality training with both image and text data.

{
        "question": "If , then x=______.", 
        "answer": "23600", 
        "message": "[{\\"role\\": \\"system\\", \\"content\\": \\"Solve the question. The user asks a question, and you solves it. You first thinks about the reasoning process in the mind and then provides the user with the answer. The answer is in latex format and wrapped in $...$. The final answer must be wrapped using the \\\\\\\\boxed{} command. The reasoning process and answer are enclosed within <think> </think> and <answer> </answer> tags, respectively, i.e., <think> Since $1+1=2$, so the answer is $2$. </think><answer> The answer is $\\\\\\\\boxed{2}$ </answer>, which means assistant's output should start with <think> and end with </answer>.\\"}, {\\"role\\": \\"user\\", \\"content\\": [{\\"type\\": \\"text\\", \\"text\\": \\"If , then x=______.\\"}, {\\"type\\": \\"image\\", \\"image\\": \\"04cf92d72682f8796387e849b4bb4.png\\"}]}]", 
        "id": "k12_14583"
        }

We visualize the changes in accuracy reward and response length during our training process. As shown in the Figure 2 and Figure 3, both accuracy and length show stable increasing trends under the current training strategy. Moreover, we can observe that the accuracy reward curve suggests potential for further improvement based on its current trend.

Evaluation

Similar to the benchmarks used in MM-EUREKA, we evaluate our model on Mathvista, MathVerse, MathVision, K12, and OlympidBench.

Note that for MM-EUREKA series models, all evaluation code uses vllm for accelerated inference, so the performance may be weaker than transformer-based methods.

<aside> 💡

We find that, despite being trained exclusively on self-collected K12 data with no overlap with the data of benchmarks such as MathVista and MathVerse, our model achieves substantial performance gains on these benchmarks. This indicates that our model has good generalization ability on out-of-domain (OOD) test sets!

</aside>