Creating an Automated Pipeline for Robustness Benchmarking Report Generation

An Explainable AI Initiative

Me (Cheng Jia Xian)

As a DigiPen student pursuing a Bachelor of Science with Honours in Computer Science in Real-Time Interactive Simulation, most of my time in school was spent working with C/C++ and game engine development. Therefore, my personal goal was to explore and learn a new area of computer science and its practical applications. In my internship application, I listed S&S as my preferred CoE to intern with as I was particularly interested in how computer vision is applied in real-world scenarios.

As part of the AI Team within S&S, my primary responsibility was to support the research and development of explainable AI (XAI) Whitebox methods. Explainable AI is a field of study within AI research that aims to provide clear explanations on the decisions of AI models. AI Models are split into two categories within XAI, White-box models and Black-box models. White-box models are highly interpretable, allowing users to understand how AI models make decisions, while black-box models provide little insight into their reasoning. XAI methods usually provide outputs in different forms, such as masks, heat maps and textual information. Explainable AI was completely foreign to me until I started my internship with HTX. It was like discovering a whole new way of thinking about AI.

Project Aim: The goal of my project was to develop a pipeline that can generate a comprehensive report that will contain useful information on both quantitative/qualitative performance and limitations of a computer vision model. This pipeline serves as a simplified method for project managers to benchmark the robustness of different computer vision models, enabling them to make more informed deployment decisions.

Fig. 1. Grad-CAM, which is a XAI Technique that will generate a heatmap according to the importance of each feature that influences the decision making of a CNN Model.

Understanding model robustness with existing open-source tools

My initial task was to study and understand the various XAI methods employed in modern industries. I spent the first couple of months working with the **Quantus XAI Toolkit**, an extensive library featuring numerous XAI methods and their use cases. My focus was to analyse how the toolkit evaluates the robustness of each XAI method. The main goal was to see if we can adopt the way the library performed their evaluation on the XAI methods.

Fig. 2. Quantus ToolKit Visualisations

Robustness in this context refers to the resilience of an XAI method when facing minor or significant changes to the input of a computer vision model. For instance, if a small perturbation is applied to an input image, the XAI method should produce results that closely align with the original input. Significant deviations would indicate a lack of robustness in the method.

I began experimenting with various XAI methods and metrics by adjusting the parameters of each test and observing the differences in outcomes. We investigated how Quantus conducted its tests and discovered that the library generates a set of images with Gaussian noise applied. This noise is not incremental but instead defined by a perturbation mean and standard deviation value.

The library uses this set of perturbed images to evaluate the robustness of XAI methods by comparing the generated mask outputs. A metric is then applied to approximate the differences between indices, yielding a quantified robustness value. Metrics such as the **Local Lipschitz Estimate, [Max/Average Sensitivity](https://arxiv.org/pdf/1901.09392) and [Continuity Metric](https://arxiv.org/pdf/1706.07979.pdf). It was at this stage where I was exposed to the different XAI methods such as [Integrated Gradients](https://arxiv.org/abs/1703.01365), Saliency maps** and the different XAI methods use to tackle visual interpretability in AI decision making.

Next, we started experimenting with the perturbation techniques in the Quantus library. By generating image sets with varying perturbation standard deviations and means, we discovered that manipulating the value of standard deviation allows us to incrementally adjust the level of noise, while changing the value of the mean alters the image’s exposure.

Fig. 3. Using a sample image from ImageNet dataset to show how adjusting the perturbation mean negatively will result in the image darkening.

Fig. 4. Using a sample image from CIFAR-10 dataset to show how adjusting the perturbation standard deviation positively will result in a noisier image result

Implementing model robustness check in an end-to-end workflow

Fig. 5. Overview of the Project, the first API will generate the synthetic dataset, the second API will then generate the prediction scores, and the last API will generate the report.

The project was split into three different parts:

● Data generation

● Prediction generation

● Report generation

This project aims to simplify the process and benchmarking of computer vision models by integrating explainability features and developing robust evaluation tools.

Data Generation Phase

Using the perturbation function from Quantus, I developed a simple script to generate synthetic datasets with incremental noise and exposure levels. Each input image is perturbed step by step, with the modifications saved at each level. As a result, each input image becomes a set of images with varying degrees of perturbation. These image sets serve as input data for a model and are used to benchmark its robustness against specific forms of perturbation. A prediction script was then created to generate the model prediction results using those generated datasets.

Fig. 6.1. Examples of synthetic data that can be generated using the Noise perturbation type, starting from clean image, more noise are introduced to the image incrementally.

Fig. 6.2. Examples of synthetic data that can be generated using the Brightness perturbation type, the image starts from very dark and gets increasingly bright.

With this, we then proceeded to fine-tune the selected open-source models using the original dataset. The open-sourced models were selected based on their popularity and their ability to converge to a satisfactory result. This is the stage of the project where I got to experience and learn about fine-tuning computer vision models. Model fine-tuning typically involves using pre-trained model weights and training with a more specific set of data. In our case, we took the open-sourced pre-trained weights and fine-tuned it using the project’s training dataset.

Prediction Generation Phase

Once the models were fine-tuned, we used the synthetic dataset that was generated to test each of the models’ capabilities and understand their limitations. Using the performance data from all fine-tuned open-source models on the identical task, we established a benchmarking range. The target model’s results were then evaluated against this range to determine if it outperformed most of the existing open-source models across the various robustness tests (noise, brightness, and feature extraction), thereby indicating satisfactory performance.

The test model will be put through the same test and the results collected. A formatted CSV file located at the specified output path:

Fig. 7. Example of the .csv file that will be generated by the function. This csv will be used to track the files and their data. Image_path, Prediction_labels, Prediction_classname, Groundtruth, Confidence, and Original_Image_path. This CSV file will include only the images that are correctly classified.

Report Generation Phase

In the report generation step, we will use the information provided from the prediction generation step. At this step, the perturbation’s respective metric scores will be calculated for quantitative comparison. This is also the step where we generate the artifacts and stitch everything together to form a report.

We wanted to give a number to the different steps of perturbation to quantify the amount of noise present in an image and the perceived brightness of an image. This is where I started researching different image quality metrics and experimenting to see which metric will be the right suite for our solution. We decided to use Peak Signal-to-Noise Ratio (**PSNR) to quantify noise and [Brightness, Chroma, Hue (BCH)](https://www.researchgate.net/publication/285554890_Brightness_Calculation_in_Digital_Image_Processing)** brightness values to quantify brightness.

Brightness, Chroma, Hue (BCH)

Why is the BCH Model used instead of the other metrics? Unlike average pixel intensity, which ignores perceptual differences between colors (e.g, green appearing brighter than blue at the same intensity), the BCH model accounts for human visual perception, providing accurate representation of brightness. The BCH model quantifies image brightness by weighing each pixel’s contribution based on its color intensity and spatial location.

Fig. 8.1. Brightness Experimentations of using BCH Value to quantify the perceived brightness values of each perturbed image. The images are perturbed by adjusting their pixel mean values and each of them was assigned a mean brightness using the BCH formula. The generated images and the graph show that BCH provides a consistent measurement of exposure/mean brightness of a digital image.

Fig. 8.2. The BCH values are calculated and used to plot a graph to visualize the increase of perceived brightness value over each perturbation step.

Fig. 8.3. BCH Formula as stated by Bezryadin et al (2007). The BCH model calculates image brightness by transforming each pixel’s RGB values using a fixed matrix and computing the square root of the sum of squares of the resulting values.

Peak-Signal-to-Noise Ratio (PSNR)

Why is PSNR used instead of other metrics? PSNR is computationally lightweight and straightforward to calculate, making it ideal for scenarios where performance and speed are critical. Unlike more complex metrics such as SSIM, PSNR does not require extensive computational resources, aligning with the goal of maintaining a simple and efficient evaluation pipeline. Additionally, PSNR’s popularity in various image processing tasks further supports its suitability, as it is widely adopted for its efficiency and ease of use.

Fig. 9.1. Example of a perturbed image compared to the original image. Lower PSNR values are observed on the image on the right (red) with more gaussian noise applied to it since the perturbation is at step 10. Compared to the image on the left (green), the PSNR score is at 65.8959 at perturbation step 1 and very little noise can be seen in the perturbed image. It is also found that using two same images will result in a PSNR index score of 80

Fig. 9.2. Plot of PSNR Index against perturbed steps which means the amount of gaussian noise applied. It is observed that the PSNR value behaves as intended on images that have artificial noise added incrementally. This qualifies the PSNR index as the metric to be employed to measure the amount of noise applied onto the perturbed image for our use case

Fig. 9.3. PSNR formula defined by Fardo, F. A., Conforto, V. H., de Oliveira, F. C., & Rodrigues, P. S. (2016, May 23). A formal evaluation of PSNR as quality measurement parameter for image segmentation algorithms. arXiv.org. https://arxiv.org/abs/1605.07116

Visualisation of the model’s performance was also done to facilitate the ease of understanding for the users. This visualisation is done in the form of a radar chart, bar charts and plotted graphs using their prediction data. The range that we deduced using the prior tests will be used as a form of benchmark for the robustness of the models that are going to be tested on this pipeline.

Fig. 10. Visualisations that will be generated using the data we have collected throughout the pipeline.

XAI masks will also be generated to tackle the issue of model interpretability. These are images that are generated after going through an explanation optimisation process that will extract the features that the models used to perform their classification. There are still underlying limitations when it comes to black-box models, one of them is that some of these XAI Methods will still require access to the model’s architecture and weights for it to produce the relevant results. The other limitation is that these XAI methods are computationally expensive to run properly and generate the mask.

Fig. 11. Stitched image of noise (left)/brightness (right) perturbed cat images with their respective ShearletX Mask. The mask can be seen as getting increasingly distorted visually as more noise is introduced over time.

Final report output

Fig. 12. Report generated using all the data collected, 1st page containing the radar chart of the performance, 2nd/3rd page containing the comparison bar charts and graph plots of the different robustness test. The 4th page contains the explainable mask generated, the last few pages will contain the text explanation on how to interpret the findings and lastly the appendix. This report will serve as a guideline for project managers on their models’ robustness.

Personal Learnings

During my time working at HTX S&S CoE, I encountered numerous challenges due to my current limited knowledge of AI/ML tasks and lack of proficiency in Python. Having primarily studied computer science through C/C++ with a focus in game development, many AI-related tasks and methodologies were new to me. I experienced a steep learning curve in the first month, requiring me to adapt my coding and development practices significantly.

To overcome the hurdles faced, I actively sought clarification and asked questions whenever I encountered something unfamiliar, turning the challenges into valuable learning opportunities.

At HTX S&S, I always had readily available support, and my supervisor was always willing to clarify my doubts. The engineers at S&S were also very helpful and always happy to offer their advice.

Each time my merge request was reviewed, I saw it as a valuable learning opportunity to identify and correct my errors. For example, I learnt that some of the coding practices I had grown accustomed to using in my projects for school were suboptimal and not best practice in the field of AI/ML.

My overall experience at HTX S&S has been very positive. The steep learning curve is both a challenge and an opportunity, especially since I came from a different background, specializing in making game engines and games during the period of my studies. However, the engineers at HTX S&S were always friendly and willing to help, which made the transition smoother. The flexible work arrangement allowed me to work autonomously, giving me the freedom to take ownership of my tasks while still having support when needed.

One of the most important lessons I learned during my time at HTX is to always approach feedback with a mindset of continuous learning. In an R&D environment, staying open to feedback is crucial for growth and improvement. As many R&D tasks require reading and understanding new concepts, it is essential to be adaptable and maintain a willingness to learn.

D&D for HTX’s 5th Anniversary that I was fortunate enough to attend.

I highly encourage future interns and engineers to use your time at HTX to explore different areas and gain as much exposure as possible. Take the initiative to learn new things, ask questions, and make the most of the resources available to you. This is a great opportunity to build valuable skills and expand your knowledge.

Please leave a comment if you have any experiences that you would like to share. If you are interested in following my future endeavors, we can connect on LinkedIn: www.linkedin.com/in/cheng-jia-xian-10a064250