Data Augmentation Benchmark for Deep Learning

Currently, data augmentation techniques for images are ubiquitous in many projects and research papers. Nevertheless, these techniques are used without a reliable reference on their effectiveness. On the contrary, the approach is most of the times completely heuristic. In order to fill in this knowledge gap, this article summarizes the results of a benchmark performed to answer:

• Which data augmentation techniques are most useful under different configurations?
• Are the 2 novel data augmentation techniques useful (image overlapping and background swapping) too?

This article summarizes the work performed by me and Diego Roa for a KTH (Royal Institute of Technology, Stockholm) subject, and the code can be found here.

Experiment Setup

The benchmark focuses on the problem of image classification using Artificial Neural Networks. The benchmark is produced by combining multiple classic data augmentation (see image on top) techniques (T4 and T5 techniques are novel) :

• T0 (basic): movement and rotation of the image.
• T1 (light deformation): slight elastic distortion.
• T2 (extreme deformation): heavy elastic distortion.
• T3 (color deformation): modify RGB values.
• T4 (image overlapping): add random shapes on top of image to partly hide image features.
• T5 (background swapping): use png images to create additional images on different random backgrounds.

Over different small datasets (transformations extend the dataset by a factor of x2):

• D0: 697 motorbike images.
• D1: 150 leopard images.
• D2: 337 face images.
• D3: 158 chest x-ray images (to detect cardiomegaly in chest x-rays).
• D4: 75 google logo images.

Classified using two different ANNs architectures:

• Small 3 layer ANN: stack of 3 2D convolutional layers, each with RELU activation and a max-pooling layer on top. The ANN is completed with 2 fully connected layers, last layer being a single neuron with sigmoid activation (for binary classification). See code in *“Training a small convnet from scratch: 80% accuracy in 40 lines of code”.*
• Pre-trained InceptionV3: using ImageNet pre-trained weights and transfer learning. On top of InceptionV3 (with pre-trained, non-modifiable weights), 2 fully connected layers are added (like in three layer ANN), which will be trained using our datasets. See code in *“Using the bottleneck features of a pre-trained network: 90% accuracy in a minute”.*

Experiment Execution

Therefore, every dataset is a classification problem that is solved using each ANN architecture. The classification problem is solved twice using the same ANN, changing only the training dataset. In one solution the training dataset will be the same as the original dataset (baseline performance), and in the other it will be augmented using data augmentation techniques (their performance will be compared). The executions are repeated 5 times with random partitions of the dataset into train and test sets.

Additionally, the problem is performing binary image classification, then the positive images are the datasets presented previously, and the negative images are random images downloaded from the Internet (with the exception of D3, which already contained negative examples). The positive and negative training and testing sets are going to be balanced, thus there will be the same number of random negative images as positive images on both train and test datasets.

Results

The results obtained show that data augmentation’s performance highly depends on the classification problem, transformation and architecture used. This means that there is no uniformity or clear trend on the accuracy results for a single dataset for all the transformations applied to it and vice versa (it changes between executions too).

Nevertheless, some particular dataset-transformation combinations showed strong and clear favorable and unfavorable behavior. In the following images, these results are remarked for both ANN architectures (green means very good transformation for such dataset, orange a regularly good and red a very bad one).

Semaphore for 3 layer model (left) and InceptionV3 (right).

Regarding the novel transformations proposed in this work, T4 and T5: as it can be seen in the images, T4 and T5 on both models do not show significant departures from the behavior of the other techniques (in terms of being good or bad in some cases and not generally). Therefore, this allows us to confidently state that T4 and T5 can be useful data augmenting techniques.

Conclusion

• Which data augmentation techniques are most useful under different configurations?

Data augmentation’s performance highly depends on the classification problem, transformation and architecture used.

• Are the 2 novel data augmentation techniques useful (image overlapping and background swapping) too?

T4 and T5 can be useful data augmenting techniques, depending also on the classification problem and architecture used.

Ethical reflection on Machine Learning

To finalize, I just wanted to add two brief ethical dilemmas that everyone working on Machine Learning should reflect upon (also, if interested, see video below):

• Privacy: when using ANNs on images, many things can be learned automatically, which opens the door to, for example, an automatic surveillance of individuals on social networks. Some learned characteristics about individuals could result in clear violations of privacy.
• Historical bias: ANNs are generally trained using historical data, which can make an ANN learn historically wrong behaviors (like racism or sexism). The trained ANN would then continue applying such behaviors without considering that this perpetuates a potentially already outdated view of our society.

[embed]