Paper Insights: DIFF4SPLAT: Repurposing Video Diffusion Models for Dynamic Scene Generation

I recently came across a GitHub repository with many papers on physically plausible video generation, and I found this paper. This paper was just presented at CVPR, and I was curious whether this method was successful and could actually be used for generating dynamic 3DGS. This paper also combined several methods I have recently been interested in: 3D Gaussian Splatting, Flow Matching, and pretrained video diffusion models.

The problem this paper seems to address is generating dynamic 3D scenes from a single image.

Related Works:

When conditioning video diffusion models, the same techniques as for conditioned image synthesis are usually employed, such as Gligen, T2i-adapter, and ControlNet. Most methods don’t study explicit camera motion control; instead, they generate 2D frame sequences. Research on video diffusion conditioning for 3D Gaussian splatting has been conducted. Lyra distills implicit 3D knowledge from video diffusion models to explicit 3DGS. Feed-forward works generate dynamic point maps, but they render improperly (with holes and artifacts).

Method:

Given a single image, a text prompt, and camera poses in Plücker embedding format. The pretrained video diffusion model is conditioned on image-camera pairs and generates a 4D latent vector z. Then, a Latent Dynamic Reconstruction Model (LDRM) processes z and camera conditions to predict a deformable Gaussian field. Then, using an efficient inter-frame deformation model, the static 3D Gaussian splatting is augmented to represent dynamics. They calculate 3 rendering losses (photometric, geometric, and motion) and a flow-matching loss, and employ a progressive training strategy.

The training datasets consist of synthetic datasets and real datasets. Authors use 7 synthetic datasets with ground-truth annotations for geometry and motion with varied camera trajectories. Authors use 2 different, mostly static, real-world datasets, then use VideoDepthAnything and MegSaM to obtain metric-scale results. The authors have a total of 130,000 quality-controlled samples.

As mentioned before, the authors introduce the Latent Dynamic Reconstruction Model(LDRM). This LDRM is a pretrained video diffusion model conditioned on an image and camera poses to generate a 3D-aware latent tensor z. This latent vector is concatenated with pose tokens and passed through a transformer before being regressed by a decoder into 3D Gaussian attributes. Finally, a 3D deconvolutional layer maps 3D Gaussian attributes to source video pixels.

A static 3D scene consists of M Gaussian primitives, each characterized by a location, scaling factors, an orientation quaternion, opacity, and color features. Spatial influence of each Gaussian is defined as:

where the covariance matrix sigma is derived from the scaling factors and orientation quaternion. A deformable 3D Gaussian for each time step t has a different calculated mean and covariance matrix:

There are 4 loss components: Flow-matching, photometric, geometric, and motion loss:

Flow matching loss transports a noise distribution to the data distribution and is only applied to the DiT that generates 3D-aware latent z. The flow-matching objectives aim to train a vector field model to match the ground truth vector field model:

The LDRM and 3D Gaussian prediction head are trained on the 3 other rendering-based losses. The photometric loss is the MSE between the rendered image and the ground truth image plus an LPIPS regularization term:

The geometric loss enforces consistency between the rendered and ground-truth depth maps. Also, a total variation loss is used to enforce smoothness. The motion loss is an L2 loss between the given 3D point tracking data and CoTracker-derived point tracks:

The model is trained in 3 steps:

• First, LDRM is trained on low-resolution static scenes with photometric and geometric losses while the deformation module is frozen.
• Then, while the deformation module is frozen, high-resolution static scenes are trained.
• Finally, the entire model is fine-tuned on dynamic datasets and uses the complete loss function.

Quantative REsults

Qualatitive Results