Activation Functions: Comparative Analysis

There are many kinds of activation functions, but three of the most important are ReLU, Leaky ReLU, and sigmoid.

ReLU (Rectified Linear Unit)

Description: Outputs zero for negative inputs and the input itself for positive values.

Advantages: It helps mitigate the vanishing gradient problem, is computationally efficient, and works well in deep neural networks.

Disadvantages: Can suffer from the dying ReLU problem, where neurons output zero for all inputs and stop learning.

Leaky ReLU

Description: A modified version of ReLU that allows small negative values instead of zero.

Advantages: It addresses the dying ReLU problem by allowing small gradients for negative inputs and works well in deep networks.

Disadvantages: The small negative slope is manually set and may not be optimal for all tasks.

Sigmoid

Description: A smooth, S-shaped function that squashes input values between 0 and 1.

Advantages: Useful for binary classification problems and provides smooth gradients.

Disadvantages: Prone to the vanishing gradient problem, making training slow in deep networks. Outputs are not zero-centered, which can slow down optimization.

We are going to evaluate a multilayer perceptron built with 3 versions of the model with the between activation functions (described above) to compare the output accuracy:

Model and Training Implementation

a) Data Preprocessing:

• The dataset is preprocessed to ensure it is in the correct format for training a neural network.
• Input images are flattened and normalized to values between 0 and 1 to improve convergence during training.

# Load and preprocess the MNIST dataset
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train = x_train.reshape(-1, 28*28).astype("float32") / 255.0
x_test = x_test.reshape(-1, 28*28).astype("float32") / 255.0
y_train = to_categorical(y_train, num_classes=10)
y_test = to_categorical(y_test, num_classes=10)

b) Model Creation:

Layers include:

• Dense Layer with 512 neurons.
• Dropout Layer to reduce overfitting.
• Output Dense Layer with 10 neurons (one for each class) and softmax activation for probability outputs.

# Function to build the MLP model 
def build_mlp_model(activation_function): 
    model = Sequential() 
    model.add(Dense(512, input_shape=(28*28,))) 
    if activation_function == 'LeakyReLU': 
        model.add(LeakyReLU(alpha=0.1))  # Adding LeakyReLU activation 
    else: 
        model.add(Dense(512, activation=activation_function)) 
    model.add(Dropout(0.2)) 
    model.add(Dense(10, activation='softmax'))  # Output layer 
    return model 

c) Training and Evaluation:

• Compiling the model with an optimizer, loss function, and evaluation metrics.
• Training the model using the training data and validating on a holdout set.
• Evaluating the trained model on the test dataset.
• This separation of training logic from model creation ensures clarity and reusability.

# Function to compile, train, and evaluate the model 
def train_and_evaluate_model(activation_function): 
    model = build_mlp_model(activation_function) 
    model.compile(optimizer='adam', 
                  loss='categorical_crossentropy', 
                  metrics=['accuracy']) 
    history = model.fit(x_train, y_train, 
                        validation_split=0.2, 
                        epochs=10, 
                        batch_size=128, 
                        verbose=2) 
    test_loss, test_accuracy = model.evaluate(x_test, y_test, verbose=0) 
    return history, test_loss, test_accuracy 

# Training and evaluation of the models with 3 different activation functions
activation_functions = ['relu', 'LeakyReLU', 'sigmoid']
results = {}

for activation in activation_functions:
    print(f"\n ----------Training model with {activation} activation function --------------")
    history, test_loss, test_accuracy = train_and_evaluate_model(activation)
    results[activation] = {
        'history': history,
        'test_loss': test_loss,
        'test_accuracy': test_accuracy
    }
    print(f"Test Accuracy with {activation}: {test_accuracy:.4f}")

The experiment compares three activation functions (ReLU, LeakyReLU, and Sigmoid) based on training, validation, and test performance on the MNIST dataset.

1. ReLU (Rectified Linear Unit)

Training Performance:

• It started with an accuracy of 91.95% in the first epoch and steadily improved to 98.73% by the 10th epoch.
• Loss reduced consistently, indicating effective learning without significant overfitting.

Validation Performance:

• Validation accuracy peaked at 97.54% after 10 epochs, which is close to training accuracy.
• Validation loss showed a decreasing trend, stabilizing toward the end, with slight fluctuations (e.g., Epoch 6 and 7).

Test Accuracy:

• Test accuracy was 97.53%, demonstrating good generalization capability.

Interpretation:

• ReLU is highly effective for this task due to its simplicity and ability to avoid the vanishing gradient problem in deeper layers.
• Slight overfitting may have occurred in later epochs, as validation accuracy plateaued while training accuracy continued to improve.

2. LeakyReLU

Training Performance:

• Started slightly lower than ReLU at 90.69% accuracy in the first epoch but reached 99.06% by the 10th epoch, the highest among all activations.
• Loss decreased more rapidly than with ReLU.

Validation Performance:

• Validation accuracy was 97.79% after 10 epochs, slightly higher than ReLU.
• Validation loss stabilized earlier than ReLU, indicating better robustness.

Test Accuracy:

• Test accuracy was 97.86%, the highest among all three activation functions.

Interpretation:

• LeakyReLU performed slightly better than ReLU due to its ability to handle small gradients for negative inputs, reducing neuron “death.”
• The model was less prone to overfitting compared to ReLU, as validation and test accuracies closely matched training performance.

3. Sigmoid

Training Performance:

• Started at 88.76% accuracy in the first epoch, slower than both ReLU and LeakyReLU, but improved to 98.41% after 10 epochs.
• Training loss decreased steadily but at a slower rate than ReLU and LeakyReLU.

Validation Performance:

• Validation accuracy reached 97.43%, slightly lower than both ReLU and LeakyReLU.
• Validation loss decreased consistently but remained higher than the other two functions.

Test Accuracy:

• Test accuracy was 97.44%, slightly lower than ReLU and LeakyReLU.

Interpretation:

• Sigmoid is less efficient for deep networks because of the vanishing gradient problem, which slows convergence.
• While it reached high accuracy, it required more epochs and showed slightly lower generalization performance than ReLU and LeakyReLU.

Overall Observations

• LeakyReLU: Best overall performance in terms of test accuracy and robustness.
• ReLU: Close second, with slightly higher overfitting potential.
• Sigmoid: Performed well but lagged slightly behind due to slower convergence and less effective gradient propagation.

References or Further reading…

Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. MIT Press.

TensorFlow. (n.d.). ReLU activation function. TensorFlow Documentation. https://www.tensorflow.org/api_docs/python/tf/nn/relu

He, K., Zhang, X., Ren, S., & Sun, J. (2015). Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. Proceedings of the IEEE International Conference on Computer Vision (ICCV), 1026–1034

*Russell, S. J., & Norvig, P. (2020). Artificial intelligence*: A modern approach (4th ed.). Pearson Education.