A Simple Introduction to Structural Similarity Index(SSIM)

Introduction

Measuring how similar two images are is a fundamental problem in computer vision, image processing, and deep learning. Whether we are reconstructing images, training generative models, performing super-resolution, or evaluating compression quality, we need a reliable way to quantify image similarity.

Traditional metrics such as Mean Squared Error (MSE) and Peak Signal-to-Noise Ratio (PSNR) have been widely used for this purpose. However, these metrics often fail to align with how humans actually perceive image quality. Two images that look almost identical to the human eye can receive a poor score, while visually degraded images can sometimes score surprisingly well.

To address this mismatch between numerical error and human perception, the Structural Similarity Index (SSIM) was introduced. SSIM focuses on comparing structural information rather than raw pixel differences, making it a perceptually meaningful image quality metric.

This article provides a clear introduction to SSIM: what it is, why it is used, how it differs from traditional metrics, and the mathematical intuition behind its formulation.

Output of using below code in a image to generate the target image from the random noise. Source of original image: https://geeksandgamers.com/wp-content/uploads/2019/02/SpiritedAway15thAnniversary_FathomEventsTrailer.jpg

What Is SSIM?

The Structural Similarity Index (SSIM) is a perceptual metric designed to measure the similarity between two images by comparing their luminance, contrast, and structural information.

Unlike pixel-wise error metrics, SSIM does not treat each pixel independently. Instead, it evaluates how local patterns of pixels relate to one another, which better reflects how humans judge visual similarity.

SSIM typically produces a value in the range:

• 1.0 → identical images
• 0.0 → no structural similarity
• Negative values → strong structural dissimilarity in theory(rare in practice, as I haven’t seen it yet)

In most image processing tasks, higher SSIM values indicate better perceptual quality.

What SSIM Measures

SSIM decomposes image similarity into three complementary components:

1. Luminance similarity — compares average brightness
2. Contrast similarity — compares variations in intensity
3. Structural similarity — compares the correlation between local patterns

This decomposition allows SSIM to focus on relative relationships within the image instead of absolute pixel values.

Why Is SSIM Used?

Limitations of Pixel-Wise Metrics

Metrics like MSE and PSNR compute error by comparing pixel intensities directly. While mathematically simple, this approach introduces several problems:

• Extremely sensitive to small shifts or misalignments
• Penalizes harmless brightness or contrast changes
• Poor correlation with perceived visual quality

As a result, optimizing solely for pixel error often produces images that look blurry or unnatural.

Practical Use Cases

SSIM is commonly used in:

• Image denoising and restoration
• Super-resolution and image reconstruction
• Generative models and autoencoders
• Image compression evaluation
• Deep learning loss functions for perceptual optimization

Your implementation and optimization loop demonstrate exactly this use case: improving perceptual similarity rather than minimizing raw pixel error.

Mathematical Intuition

The SSIM between two image patches x and y is defined as:

SSIM Equation

SSIM is computed locally, typically using a Gaussian-weighted window.

Local Mean (Luminance)

Local Mean captures the average brightness in a neighborhood.

Intuition: This captures average brightness in a neighborhood, not globally.

Local Variance (Contrast)

Local variance captures the local contrast or texture strength

Intuition: Variance measures local contrast or texture strength.

Local Covariance (Structure)

Local covariance captures the local pattern which align between two images

Intuition: Covariance measures how well local patterns align between images.

Decomposed SSIM Form

SSIM can be written as the product of three components:

Decomposed SSIM into its main components, each measuring a different aspect of image similarity, namely luminance, contrast, and structural similarity.

Stability Constants

The constants are defined as:

where:

• L is the dynamic range of pixel values
• k1≈0.01, k2≈0.03

Purpose:

• Prevent division by zero
• Stabilize SSIM in flat or low-contrast regions

How SSIM Differs From Other Image Quality Metrics

SSIM vs Mean Squared Error (MSE)

MSE computes the average squared difference between corresponding pixels. While easy to optimize, it treats all errors equally, regardless of perceptual importance.

Key differences:

• MSE is pixel-centric
• SSIM is structure-centric
• MSE ignores spatial relationships
• SSIM explicitly models them

Two images with identical structures but slightly different brightness may have high MSE but high SSIM — which better matches human perception.

Demonstration of a distorted image with a similar MSE, but when we observe it through our perception, it looks different, which is captured by SSIM. Source: https://www.cns.nyu.edu/~lcv/ssim/

Why SSIM Works Well as a Loss Function

SSIM provides gradients that encourage models to preserve:

• Edges
• Local contrast
• Spatial coherence

Unlike L1 or L2 loss, SSIM penalizes perceptually harmful distortions more strongly than harmless pixel-level changes. This often results in sharper, more natural reconstructions — as your optimization experiment clearly demonstrates.

Code

Code for demonstrating the image reconstruction from random noise to target image, using SSIM:

# Requirements are pillow, torch, numpy, matplotlib
import torch
import numpy as np
import torch.nn as nn
import torch.nn.functional as F

from PIL  import Image
from matplotlib import pyplot as plt

class SSIM(nn.Module):
    def __init__(
        self,
        window_size: int = 21,
        sigma: float = 1.5,
        data_range: float = 1.0,
        k1: float = 0.01,
        k2: float = 0.03,
        reduction: str = "mean"
    ):
        super(SSIM, self).__init__()
        self.window_size = window_size
        self.sigma = sigma
        self.data_range = data_range
        self.k1 = k1
        self.k2 = k2
        self.reduction = reduction

        # Create Gaussian kernel, registered as buffer for device moment
        kernel = self._create_gaussian_kernel(window_size, sigma)
        self.register_buffer('kernel', kernel)

        # Constants
        self.C1 = (k1 * data_range) ** 2
        self.C2 = (k2 * data_range) ** 2

        self.pad = window_size // 2

    def _create_gaussian_kernel(self, window_size: int, sigma: float):
        coords = torch.arange(window_size, dtype=torch.float32)
        coords -= window_size // 2
        g = torch.exp(-(coords ** 2) / (2 * sigma ** 2))
        g = g / g.sum()

        kernel_2d = g.unsqueeze(0) * g.unsqueeze(1)   # [window_size, window_size]
        kernel = kernel_2d.unsqueeze(0).unsqueeze(0)  # [1, 1, window_size, window_size]
        return kernel

    def forward(self, img1: torch.Tensor, img2: torch.Tensor) -> torch.Tensor:
        if img1.shape != img2.shape:
            raise ValueError(f"img1 and img2 must have same shape, got {img1.shape} and {img2.shape}")

        batch, channel, _, _ = img1.shape
        kernel = self.kernel.expand(channel, 1, self.window_size, self.window_size)

        # Move kernel to same device/dtype as input
        kernel = kernel.to(device=img1.device, dtype=img1.dtype)

        # Local means
        mu1 = F.conv2d(img1, kernel, padding=self.pad, groups=channel)
        mu2 = F.conv2d(img2, kernel, padding=self.pad, groups=channel)

        mu1_sq = mu1 ** 2
        mu2_sq = mu2 ** 2
        mu1_mu2 = mu1 * mu2

        # Local variances and covariance
        sigma1_sq = F.conv2d(img1 * img1, kernel, padding=self.pad, groups=channel) - mu1_sq
        sigma2_sq = F.conv2d(img2 * img2, kernel, padding=self.pad, groups=channel) - mu2_sq
        sigma12 = F.conv2d(img1 * img2, kernel, padding=self.pad, groups=channel) - mu1_mu2

        ## SSIM map
        ssim_map = ((2 * mu1_mu2 + self.C1) * (2 * sigma12 + self.C2)) / \
                    ((mu1_sq + mu2_sq + self.C1) * (sigma1_sq + sigma2_sq + self.C2))

        if self.reduction == "mean":
            return ssim_map.mean()
        elif self.reduction == "none":
            return ssim_map.mean(dim=[1, 2, 3])
        else:
            raise ValueError("reduction must be 'mean' or 'none'")

if __name__ == "__main__":
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    print(f"Using device: {device}")

    ssim = SSIM(data_range=1.0, reduction="none").to(device)
    image_path = "./sample_image.png"

    image = Image.open(image_path).convert("RGB")
    image = np.asarray(image).astype(np.float32)
    # normalize the image
    image = image / 255.0
    target = torch.from_numpy(image).permute(2, 0, 1).unsqueeze(0).to(device)
    _, _, h, w = target.shape
    img = torch.rand(1, 3, h, w, device=device, requires_grad=True)

    optimizer = torch.optim.Adam([img], lr=1e-2)

    plt.ion()
    fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(10, 5))
    STEPS = 500

    for step in range(STEPS):
        optimizer.zero_grad()

        # Compute loss (1 - SSIM)
        ssim_value = ssim(img, target)
        loss = 1.0 - ssim_value
        loss.backward()
        optimizer.step()

        # Clamp to valid pixel range
        with torch.no_grad():
            img.clamp_(0, 1)

        if step % 20 == 0:
            ax1.clear()
            ax1.imshow(target[0].cpu().permute(1, 2, 0))
            ax1.set_title("Target")
            ax1.axis("off")

            ax2.clear()
            ax2.imshow(img[0].detach().cpu().permute(1, 2, 0))
            ax2.set_title(f"Step {step} | SSIM: {ssim_value.item():.4f}")
            ax2.axis("off")

            plt.pause(0.01)
    plt.ioff()
    plt.show()

There is an error in the final generated image compared to the target image, even when SSIM = 1. When we subtract the generated image from the original target image, we obtain the “original error image,” which contains very narrow pixel differences and is therefore difficult to visualize. To make these differences more visible, I stretched the pixel values for better visualization. This demonstrates that even though the generated image and the target image have SSIM = 1 (indicating full similarity from a perceptual standpoint), they are not identical at the pixel level. Alternatively, I could have used a grayscale version instead of contrast stretching to observe the differences, but I simply wanted to experiment with this approach 🙂

Conclusion

SSIM is a powerful image similarity metric designed to bridge the gap between numerical accuracy and human perception. By focusing on luminance, contrast, and structure, it provides a more meaningful assessment of visual quality than traditional pixel-wise metrics.

For tasks where perceptual quality matters — especially in modern deep learning workflows — SSIM is often a superior choice.

References:

• Original code in gist: https://gist.github.com/realsanjeev/b8e68ea5246e96527c01411ac1fad4fe
• Structural Similarity Index Measure (SSIM). Wikipedia. https://en.wikipedia.org/wiki/Structural_similarity_index_measure
• SSIM — scikit-image Documentation. https://scikit-image.org/docs/stable/auto_examples/transform/plot_ssim.html
• The SSIM Index for Image Quality Assessment. CNS, New York University — includes clear visual illustrations of SSIM. https://www.cns.nyu.edu/~lcv/ssim/
• Understanding SSIM. arXiv preprint. https://arxiv.org/abs/2006.13846