A Unified Video-Action Model: How UVA Folds Video Generation and Action Prediction Into One

A paper out of Shuran Song’s group at Stanford, UVA (Unified Video Action Model), merges “video generation” and “action prediction” into a single model. At inference time, the predicted actions are applied directly to a robot (an ARX X5 arm with a two-finger gripper).

The motivation starts with a simple observation: video carries rich scene and physical information that helps with predicting actions, and conversely, actions carry dynamic information about how the environment changes, which helps with predicting video. In theory, the two can complement each other to some degree.

But earlier approaches generated the video first and then predicted actions from it, which was both slow and not especially effective. You had to roll out future frames one by one, with inference dragging out to twenty-plus seconds, and errors in the video would propagate all the way down to the actions. The result was that accuracy actually lost out to directly learning a policy with Diffusion Policy.

UVA’s solution has two key pieces. The first is learning a joint video-action latent representation: past observations and past actions are fed together through a Transformer and fused into a shared set of latent tokens Z, from which both future frames and future actions are decoded. So this latent space holds information from both the visual and the action sides at once.

The second is decoupling the video and action decoding, using two very lightweight diffusion heads — one decoding video, one decoding action — so that denoising runs only on these two light heads rather than across the entire large network. This way, during training, video generation pushes the latent Z to capture finer scene detail, which in turn helps the action side.

When you actually deploy the policy, you simply skip the video-generation branch and decode only the action. Inference speed is essentially on par with Diffusion Policy, and two orders of magnitude faster than UniPi (24 seconds), which genuinely generates video.

Training is done via masked learning (Figure 1). The same model can act as a policy, a video generator, a forward or inverse dynamics model, and even as a policy-plus-planner combined — several tasks unified under one framework. It can also be pretrained from pure video data with no action annotations.

Experimentally, UVA matches or surpasses the purpose-built methods across seven benchmarks overall, and it’s especially strong in multi-task settings: 20 points higher than the best baseline on multi-task PushT, and 5 points higher on Libero10.

| Find papers faster on arXivSub with AI summary (CVPR/ICCV/ICML/ICLR/NeurIPS/AAAI/MICCAI)