Mixup for better generalization of neural networks.

What is Mixup ?

A data augmentation technique involving random convex combination of raw inputs, and correspondingly, convex combination of one-hot label encodings.

(convex combination is a special type of linear combination of 2 or more vectors in which coefficients are non-zero and add up to 1.)

In the mixup method virtual training examples are constructed using the following equation-

(xi , yi) and (xj , yj ) are two samples that are drawn at random from the training data, and λ ∈ [0, 1].

Photos by Old Youth and Chris Andrawes on Unsplash

The need

Most popular neural network architectures work on the principle of Empirical Risk Minimization (ERM) and their size varies linearly with the size of the training samples.

But there are some key issues with the ERM principle.

• Memorization of the training data: ERM allows large neural networks to memorize (instead of generalize from) the training data even in the presence of strong regularization, or in classification problems where the labels are assigned at random (Zhang et al., 2017).
• Adversarial example: Neural networks trained with ERM change their predictions drastically when evaluated on examples just outside the training distribution ,also known as adversarial examples.

Data Augmentation — an alternative to ERM

Data augmentation is commonly used in image classification, where the vicinity of one image is defined as the set of its horizontal flips, shear, rotations and blurring, to name a few. Additional similar virtual examples can be drawn from this vicinity distribution of the training examples to augment the training distribution. This results in considerable improvement in model generalization. The main drawback of this technique is that it needs domain expertise to define the vicinity and is also not data agnostic.

Better option is mixup and let us explore some math to understand how it works under the hood.

Our objective in supervised learning tasks is to identify a function f ∈ F that explains the relationship between a random feature vector X and a random target vector Y , which follow the joint distribution P(X, Y ).

The first step is to define a loss function L that penalizes the differences between predictions f(x) and the actual target y, for training examples (x, y) following distribution P. In the next step we minimize the average of this loss function over the data distribution P, also known as the expected risk:

So far so good. Unfortunately, this distribution P is unknown in most practical situations and we have to approximate P using the training data D. We can approximate the distribution P in the Vicinal Risk Minimization principle (VRM) as-

where ν is a vicinity distribution that measures the probability of finding the virtual feature-target pair in the vicinity of the training feature-target pair .To learn using VRM, we sample the vicinal distribution to construct a dataset and minimize the empirical vicinal risk:

The mixup vicinal distribution can be understood as a form of data augmentation that encourages the model f to behave linearly in-between training examples.

Mixup: a generic vicinal distribution contributed by facebook AI research

where λ ∼ Beta(α, α), for α ∈ (0, ∞).

In a nutshell, sampling from the mixup vicinal distribution produces virtual feature-target vectors :

where (xi , yi) and (xj , yj ) are two feature-target vectors drawn at random from the training data, and λ ∈ [0, 1].

The mixup interpolation α, is a hyper parameter that controls the strength of interpolation between feature-target pairs.

Python Code

We can implement mixup using a few lines of code and without any computation overhead.

[embed]

Benefits

• Significant improvement in the generalization with image datasets such as CIFAR-10, CIFAR-100, and ImageNet-2012.
• Increases the robustness of the neural network to incorrect labelled data and adversarial samples .
• Improves the training process for generative adversarial networks.
• Increases the generalization on tabular data as well as speech.
• can be implemented easily into existing pipelines of training data and the there is a minimal increase in the computational load.

Suggested implementation strategy

Experiments conducted at Facebook AI research showed that using a single data loader to generate a mini batch of data, random shuffling and application of mixup yielded good results and also kept the I/O requirements under check.

During training the models with mixup demonstrated two key characteristics-

• Fewer prediction errors. On a ResNeXt-101 model the mix-up model with alpha=5 trained over 90 epochs had a top-5 error of 4.9 opposed to 5.3 for a ERM model trained over 100 epochs.
• The norm of the gradients of the model with respect to input in-between training data was smaller. In other words the loss function landscape is flatter, which results in better model generalization (Zhao et al., 2022)

Mixup Use Cases

1. Image Classification: The models trained with mixup demonstrated better performance over the image datasets CIFAR-10 and CIFAR-100. On a DenseNet-BC-190 the mixup model test errors lower than ERM model by 2.2 points.
2. Speech recognition: Using the Google commands dataset, classification errors with a VGG-11 were lower than ERM when mixup was applied to the sound waveforms at the spectrogram level.
3. Adversarial Examples: Adversarial examples are generated by adding tiny (visually imperceptible) perturbations to valid examples in order to degrade the model performance. Improving the robustness to adversarial noise is an active research area. In the research by Zhang et al. (cited below) mixup trained neural networks proved to be more robust than ERM against adversarial examples in white box and black settings without any computational overhead.
4. Memorization of corrupted labels: The resistance to noisy labels in the training data shows significant improvement when mixup technique is combined with dropout. This confirmed that the two methods are compatible and be leveraged together to reduce training error.
5. Generative Adversarial Network training stability: In GANs, a generator and a discriminator compete against each other to model a distribution P. Common challenge in solving the optimization problem in GANs is vanishing gradients supplied by the discriminator to the generator. The use of mixup in GAN training has shown to regularize the gradients of the discriminator that ensures a stable source of gradient information to the generator.

Mixup formulation of GANs

Mixup design choices

Some interpolation choices we can play with when designing the mixup-

• Interpolation of latent representations (i.e. feature maps) of a neural network.
• Interpolation only between the nearest neighbors using KNN.
• Interpolation only between inputs belonging to the same class (SC) or all classes(AC).

Conclusion

Mixup is a data-agnostic and simple data augmentation technique based on vicinal risk minimization that involves creation of virtual examples constructed as the linear interpolation of two random examples from the training set and their labels.The mixup technique has been quite effective in improving model generalization and minimize memorization of corrupt labels, reduce sensitivity to adversarial examples, and combat instability in adversarial training.

Try out the Mixup here: https://github.com/Roopg/Mixup_technique/blob/main/Mixup_data_augmentation.ipynb

References

mixup: BEYOND EMPIRICAL RISK MINIMIZATION https://arxiv.org/pdf/1710.09412.pdf

@article{ zhang2018mixup, title={mixup: Beyond Empirical Risk Minimization}, author={Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, David Lopez-Paz}, journal={International Conference on Learning Representations}, year={2018}, url={https://openreview.net/forum?id=r1Ddp1-Rb}, }