(Paper Review)One-for-More: Continual Diffusion Model for Anomaly Detection

This post reviews a paper “One-for-More: Continual Diffusion Model for Anomaly Detection (CVPR’25)”, which detects anomalies using a diffusion model.Goal

This paper aims to inspect defects in images by leveraging diffusion models. Common anomaly detection methods based on diffusion typically perform one-to-one anomaly detection, meaning they can detect only a single type of trained anomaly. These methods cannot be easily extended to detect multiple types of anomalies. Therefore, this paper proposes an extendable anomaly detection framework that can continually expand to include new types of anomalies.

our goal is to design a continuously updated diffusion model via task-by-task training to generate or reconstruct various kinds of normal images and then measure the difference between the generated normal image and the test image, to determine whether there exists a defect.

Continual anomaly detection involves addressing two key challenges:

(1) Catastrophic Forgetting [link]: Catastrophic forgetting refers to the phenomenon where training on new tasks causes the model to forget previously learned tasks.

when one incremental step has many classes, it will still cause a large forgetting of the previous tasks.

For example, in the figure below, ControlNet shows a significant drop in A-AUROC as the number of trained tasks increases.

Figure from the original paper.

(2) Faithfulness Hallucination: Hallucination occurs when the model generates an image belonging to a different class. The figure below illustrates an example of hallucination in ControlNet. ControlNet is supposed to generate an image resembling a “black circle,” but instead, it produces an image from another class.

Figure from the original paper.

Key Features

This paper proposes a Continual Diffusion model for Anomaly Detection (CDAD) by introducing the following methods:

(1) Gradient Projection: To train new tasks without “Catastrophic Forgetting”, the authors introduce gradient projection, which projects the update gradients in a direction orthogonal to the gradients of previous tasks.

In this paper, we propose a continual diffusion model framework that projects the updating gradient of new tasks onto a subspace orthogonal to the significant representation of previous tasks [29, 34], thus eliminating the effect of the update gradient on the previous feature space.

(2) Iterative Singular Value Decomposition(iSVD): Since gradient projection requires large memory, the authors propose iSVD.

Since the significant representation is calculated through singular value decomposition (SVD) on the matrix unfolded by intermediate features, the gradient projection operation requires a lot of memory.

To address this issue, we propose an iterative singular value decomposition (iSVD) method based on linear transitivity theoretical property.

(3) Anomaly-masked Network(AMN): o prevent hallucination, the authors introduce AMN.

Another issue of DM-based anomaly detection methods [13, 48] is that these methods tend to “over-fitting” normal samples rather than focusing on “reconstructing” anomalous regions, which weakens the role of the image-to-image conditioning mechanism and aggravates the hallucination problem in continuous learning. To this end, we propose the anomaly-masked network.

This post omits detailed mathematical expressions and instead provides a simplified explanation of the main concepts of each method. For more details, please refer to the original paper.

Gradient Projection

The key idea to address the challenge of catastrophic forgetting is to keep the previous feature space unchanged after each gradient update. To achieve this, the weight updating process follows the steps below:

Figure from the original paper.

(1) Project∇w onto the orthogonal space of span(X).

(2) To obtain this orthogonal space, compute the k-rank column basis U.

(3) By projecting∇w onto U with K-rank approximation, the model can be trained while retaining the previously learned knowledge.

The updated model on the new task does not perturb the output space of the previous task, thus ensuring the model retains the old knowledge.

Iterative Singular Value Decomposition(iSVD)

The Singular Value Decomposition(SVD) used to compute U requires a global calculation causing memory overhead. To reduce memory usage, the authors propose Iterative Singular Value Decomposition(iSVD).

Figure from the original paper.

(1) Split the large matrix M into smaller sub-matrices m.

(2) For each small matrix m, compute U by k-rank approximation.

(3) Concatenate all U spaces obtained from each sub-matrix.

Lemma 1 linear transitivity: Given matrices S1, S2, and S3, if S1 can be linearly expressed by S2, and S2 can be linearly expressed by S3, then S1 can be linearly expressed by S3.

Table from the original paper. iSVD significantly reduces memory usage.

Anomaly-masked Network(AMN)

To address two main issues (“identical shortcuts” and “overfitting”) in anomaly detection using image-to-image diffusion, the authors introduce AMN as shown in the below figure.

Figure from the original paper.

(1) Local Encoder is CNNs with SiLU activation. It transforms the input data into the input shape of U-net.

(2) Non-local Encoder is composed of the transformer encoder layer with neighbor-masked self-attention(NMSA).

(3) Patch perturbation is applied in Non-local encoder to mask out abnormal region.

Results & Conclusion

The authors conducted extensive experiments on the MVTec and VisA datasets. The main experiment evaluates the A-AUROC score while gradually increasing the number of anomaly classes. However, the explanation of the experimental settings is somewhat insufficient (it only refers to the IUF paper). Nevertheless, the experimental results demonstrate that CDAD achieves significant performance when trained on multiple classes by effectively overcoming the catastrophic forgetting problem. If you are interested in the experimental results, I recommend reading Section 4.1 of the original paper.