Building a Neural Network for Binary Classification from Scratch: Part 3 (From Training to Evaluation )

Caterpillar

Building neural networks from scratch is an exciting way to truly understand how they work. In this final part, we’ll train our binary classification network.

In **Part 1, we prepared the MNIST dataset and designed the network architecture. In [Part 2](https://medium.com/@abhiveerhome/building-a-neural-network-for-binary-classification-from-scratch-part-2-the-math-behind-neural-36d5aaa1acb0), we explored the core mechanics, including the forward pass, cost function, backpropagation, and gradient descent. Now, in Part 3**, we’ll bring it all together — training the model, saving it, and visualizing its results.

You can refer to code here

Training the Neural Network

Training is the process of optimizing the network’s weights and biases to minimize error. Here’s how it works:

1. Feed the dataset through the network: Pass the input data through the network to generate predictions.

2. Calculate predictions and measure the error (cost): Use a cost function (e.g., binary cross-entropy) to calculate how far the predictions are from the actual values.

3. Adjust weights and biases to reduce the error: Use backpropagation to compute gradients and update weights and biases with gradient descent.

4. Repeat for multiple iterations (epochs): Iterate over the entire dataset multiple times to continuously improve the model’s accuracy

Code for Training

We train the network using our gradient descent implementation:

# Train the neural network
W1, b1, W2, b2, W3, b3, training_time = gradient_descent(
    x_train_filtered, y_train_filtered, W1, b1, W2, b2, W3, b3,
    activation_hidden="relu", learning_rate=0.01, num_epochs=5000
)

• Learning Rate (α): Controls the size of weight updates; α = 0.01.
• Epochs: The model trains over 5000 iterations of the entire dataset, repeatedly updating the weights.

Training Progress

As training progresses, the cost decreases, indicating the network is learning.

Saving the Model

After training our neural network, we save its weights and biases in the form of binary .npy files. These files act as the final representation of our trained model. Here's why and how we do it:

Why Save the Model?

1. Reusability: Once the network is trained, saving the weights and biases allows us to reuse the model without retraining it.
2. Efficiency: Avoids the need for lengthy training processes every time we want to make predictions.
3. Portability: The model can be loaded and used on any compatible machine or system.

How We Save the Model?

We use NumPy’s save() function to store the weight and bias matrices into .npy files. For example:

np.save("models/W1.npy", W1)
np.save("models/b1.npy", b1)
np.save("models/W2.npy", W2)
np.save("models/b2.npy", b2)
np.save("models/W3.npy", W3)
np.save("models/b3.npy", b3)

Each file stores the corresponding matrix or vector in a compact binary format.

How We Load the Model?

Later, when we need to use the model for predictions, we simply load these files using NumPy’s load() function:

W1 = np.load("models/W1.npy")
b1 = np.load("models/b1.npy")
W2 = np.load("models/W2.npy")
b2 = np.load("models/b2.npy")
W3 = np.load("models/W3.npy")
b3 = np.load("models/b3.npy")

Addressing Challenges

1. Overfitting and Underfitting

• Overfitting: The model memorizes the training data but performs poorly on unseen data.
• Underfitting: The model fails to capture patterns in the data.

Illustrates how an overfitted model tries to fit the training data exactly, leading to poor generalization.

Solutions:

• Overfitting: Use regularization or dropout (future enhancement).
• Underfitting: Increase the model’s complexity or train for more epochs.

Overfitting often occurs when models are overly complex or trained for too long. Techniques like regularization and dropout, which we’ll explore in future articles, help address this

2. Learning Rate Challenges

Choosing the wrong learning rate can hinder training:

• Too Small: Learning is painfully slow.

Slow reduction in cost due to a small learning rate, indicating inefficiency in training.

• Too Large: The updates are unstable, causing oscillations.

rge learning rate causes oscillations, preventing the model from converging.

Evaluating the Neural Network

After training, we need to measure how well the network performs. Here’s how we evaluate it:

1. Model Summary

The model_summary() function provides key insights into the trained network, including:

• Total parameters (weights and biases).
• Training time.
• Cost reduction.
• Accuracy on the training dataset.

def model_summary(W1, b1, W2, b2, W3, b3, training_time, initial_cost, final_cost, learning_rate, num_epochs, activation_hidden, init_method,accuracy):
    # Calculate total parameters
    total_parameters = (W1.size + b1.size) + (W2.size + b2.size) + (W3.size + b3.size)
    cost_reduction = ((initial_cost - final_cost) / initial_cost) * 100  # Percentage reduction

    print("\nModel Summary:")
    print(f"Number of Training Images: {num_images}")
    print(f"Number of '0' Images: {num_zeros}")
    print(f"Number of '1' Images: {num_ones}")
    print(f"Input Image Shape: (784,) (Flattened from 28x28)")
    print("Normalization: Pixel values scaled to [0, 1]\n")

    print("Training Configuration:")
    print(f"- Activation Function (Hidden Layers): {activation_hidden.capitalize()}")
    print(f"- Activation Function (Output Layer): Sigmoid")
    print(f"- Weight Initialization: {init_method.capitalize()}")
    print(f"- Learning Rate: {learning_rate}")
    print(f"- Number of Epochs: {num_epochs}")
    print(f"- Initial Cost Value: {initial_cost:.4f}")
    print(f"- Final Cost Value: {final_cost:.4f}")
    print(f"- Cost Reduction: {cost_reduction:.2f}%")
    print(f"- Accuracy: {accuracy:.5f}%\n")
    print(f"- Training Time: {training_time:.2f} seconds\n")

    print("Weight and Bias Parameters:")
    print(f"- W1: {W1.shape}, b1: {b1.shape}")
    print(f"- W2: {W2.shape}, b2: {b2.shape}")
    print(f"- W3: {W3.shape}, b3: {b3.shape}")
    print(f"- Total Parameters: {total_parameters}\n")

    print("Environment Details:")
    print("- Hardware: CPU")  # Update this if you use GPU in the future
    print("- Python Version:", sys.version)
    print("- NumPy Version:", np.__version__)

model_summary(W1, b1, W2, b2, W3, b3, training_time, initial_cost, final_cost, learning_rate, num_epochs, activation_hidden, init_method, accuracy)

Model Summary:
Number of Training Images: 12665
Number of '0' Images: 5923
Number of '1' Images: 6742
Input Image Shape: (784,) (Flattened from 28x28)
Normalization: Pixel values scaled to [0, 1]

Training Configuration:
- Activation Function (Hidden Layers): Relu
- Activation Function (Output Layer): Sigmoid
- Weight Initialization: X
- Learning Rate: 0.001
- Number of Epochs: 10
- Initial Cost Value: 0.6917
- Final Cost Value: 0.6900
- Cost Reduction: 0.24%
- Accuracy: 54.47296%

- Training Time: 0.34 seconds

Weight and Bias Parameters:
- W1: (784, 25), b1: (1, 25)
- W2: (25, 15), b2: (1, 15)
- W3: (15, 1), b3: (1, 1)
- Total Parameters: 20031

Environment Details:
- Hardware: CPU
- Python Version: 3.11.6 (main, Apr 10 2024, 17:26:07) [GCC 13.2.0]
- NumPy Version: 2.0.2

2. Accuracy Calculation

We implemented a custom accuracy function to measure how many predictions are correct:

def calculate_accuracy(x_train, y_train, W1, b1, W2, b2, W3, b3, activation_hidden):
    _, _, a3 = forward_pass(x_train, W1, b1, W2, b2, W3, b3, activation_hidden, "sigmoid")
    predictions = (a3 > 0.5).astype(int)  # Convert probabilities to binary output
    correct_predictions = np.sum(predictions == y_train)
    accuracy = (correct_predictions / y_train.shape[0]) * 100  # Calculate accuracy percentage
    return accuracy

#Usage:
accuracy = calculate_accuracy(x_train_filtered, y_train_filtered, W1, b1, W2, b2, W3, b3, activation_hidden="relu")
print(f"Training Accuracy: {accuracy:.2f}%")

Conclusion🍺

Over the last three articles, we’ve journeyed through the entire lifecycle of building a neural network from scratch:

1. Prepared the MNIST dataset and designed a neural network architecture (Part 1).
2. Explored the inner workings of neural networks, including forward pass and backpropagation (Part 2).
3. Trained the network, evaluated its performance, and visualized its predictions (Part 3).

What’s Next?

We’ll push our network to classify all 10 digits (0–9), expanding its capabilities and refining its architecture.

To build on these concepts, we’ll soon explore multi-class classification in a new 3-part series. This next journey will cover:

1. Preparing the dataset for recognizing digits 0–9.
2. Designing a scalable neural network architecture and the maths behind it.
3. Training the network and analyzing its performance across all 10 classes.

Final Notes

This series highlighted the power of building neural networks from scratch, helping you understand each step, from data preparation to training and evaluation. The “no external libraries” challenge showcased how fundamental components come together to create a working neural network.

It’s your turn — experiment with the code, tweak the parameters, and see how the network behaves!