NeurIPS 2025 oral |New SOTA for infrared and visible image fusion (IVIF)

I was browsing the NeurIPS 2025 list over the last couple of days and came across this paper on IVIF (Infrared and Visible Image Fusion). The approach seems quite interesting. This field focuses on overlaying images from two types of light, allowing you to clearly see environmental contours and details while also spotting thermal targets within the scene (such as people, cars, and animals).

Existing SOTA (State-of-the-Art) methods score very high on clean datasets, but as soon as they encounter real-world rain, fog, low light, or noise, their performance takes a massive hit, and artifacts often appear.

This paper tackles two key issues: “Real-Scene Degradation” and “User Controllability.”

1️⃣ First, it addresses the often-overlooked source data problem.

Many current methods are trained on simple synthetic data, which differs significantly from the real world. Real situations involve different weather conditions, lighting, and even camera shake.

The authors constructed a physics-driven degradation imaging model to augment the data. It is simulated based on Retinex theory (a psychophysical theory of “how the human eye sees”) and atmospheric scattering principles. It accounts for rain and fog in visible light, and also simulates stripe noise and low contrast for infrared images.

This is actually quite important; for tasks intended for real-world environments, physical priors remain key to improving generalization capabilities.

2️⃣ Second is the architecture: It proposes a Prompt-based network.

It introduces CLIP’s text encoder, allowing you to control the fusion process using natural language. For example, you can explicitly specify “Visible light has fog, Infrared has noise,” and the network will dynamically adjust feature distribution to better remove the noise.

The authors considered that during actual deployment, you can’t expect users to hand-write a prompt for every image. So, they designed a Spatial-Frequency Synergistic Visual Adapter (SFVA). This module uses frequency domain features to extract degradation clues. In other words, even if you don’t give instructions, the model can automatically perceive whether the image is blurry or overexposed through differences in the frequency domain, enabling automated processing.

Moreover, this module performs almost as well as CLIP because, during the training phase, it underwent alignment (or knowledge distillation) with CLIP, allowing the SFVA features to learn and approximate CLIP’s features.

Experiments compared it against some of the latest methods. ControlFusion pushed metrics (especially VIF and SD) to new highs on datasets like MSRS and RoadScene. Particularly in mixed degradation scenarios (e.g., low light on one side and stripe noise on the other), its recovery effect is very clean while preserving the infrared thermal targets (Figures above).

Overall, this work validates a much more valuable direction: Instead of trying to squeeze out a fraction of a percentage point in metrics by deepening networks in ideal environments, it is better to combine physical imaging mechanisms with the semantic understanding of multimodal large models to solve the underlying problems first.

| Find papers faster on arXivSub with AI summary (CVPR/ICCV/ICML/ICLR/NeurIPS/AAAI/MICCAI)