Deep Learning with PyTorch — Introduction & Designing of L-layer Neural Network using NumPy: S1 E1

Hey there! I’m thrilled to announce to share my knowledge on this series of blogs where we delve into developing applications using PyTorch. Before going into it, let me introduce myself. I’m Sumanth, a Data Scientist working at Sigmoid with 2 Years of Experience in developing machine learning, deep learning and analytical solutions for Consumer Package Industries / Fast Moving Consumer Goods (CPG / FMCG), Financial Companies. Prior to joining Sigmoid, I graduated with a Bachelor of Technology degree from Indian Institute of Technology Kharagpur. I love to code and develop applications using PyTorch and C++. I’m proficient in Python, C++, Machine Learning, Computer Vision, Algorithms and NLP. Now, we’ll go into the world of Practical Deep Learning.

I said “Practical Deep Learning” because of learning deep learning through practical datasets. In this blog, I’ll explain you how to develop and understand a Neural Network using NumPy and develop a non linear classifier to classify the data and assess its accuracy. At last, we’ll get to know about a popular deep learning framework from meta, namely PyTorch developed by Soumith Chintala’s group at Facebook in 2016. We’ll also develop a classifier to classify the data using PyTorch.

Industrial approach to develop any machine learning system includes a process of steps ( called pipelines) that would be run after one another to facilitate a streamline flow from data ingestion to model outputs and predictions and deployment on cloud. In this blog, we’ll take inspiration from Industrial approaches to design a neural network classifier. The so called pipelines are described below. We’ll go from developing the below from L-layer Neural Network in NumPy to PyTorch. The below code is available on my github

1. Data Creation / Data Fetch Pipeline
2. Understanding about Data through visualization, Distributions of Target Vector and Feature Vectors
3. Feature Engineering Pipeline
4. Defining the Neural Network Architecture
5. Model Pipeline
6. Test Pipeline
7. Deploying Pipeline

We’ll go through each and individual steps to understand the entire process. We’ll implement the same in both NumPy and PyTorch. In this blog, I’ll present you a L-layer neural network developed using NumPy that can classify whether an image is a dog or a cat. This blog clearly explains back propagation, forward propagation and cost computation using numpy.

Data Creation Pipeline

Most of the models in deep learning are data driven. So it’s important to have a good dataset which is unbiased and can be generalized well enough on both train and test sets. We make sure that Train Set and Test Set assumes same probability distribution. *Cats and Dogs* dataset is publicly available on kaggle. Download the dataset on your local computer, create a jupyter notebook / data_prep.py file in the same directory that you’ve just downloaded (I assume that you’ve created a virtual env on your system using conda / pip3). The below function creates a training_images and training_labels if mode is set to train else it creates a testing_images and ids (as described in kaggle data overview)

import time
def load_dataset(mode = "train"):
    """
    Iterates over each image in train and creates two tensors X and y which consists of Images and Labels. Images are resized into
    (64,64,3) RGB. Accepts two modes. mode = "train" creates Training Images and Labels and mode = "test" creates Transformed Test Images

    Args:
    mode: "train" or "test"

    returns:
    training_images, training_labels if mode is train or testing_images
    """
    start_time = time.time()
    if mode == "train":
        training_images = []
        training_labels = []
    else:
        testing_images = []
        ids = []
    main_dir = os.getcwd()
    mode_dir = mode
    path = os.path.join(main_dir,mode_dir)
    convert_into_label = lambda  category: int(category == 'dog') # return 1
    if mode == "train":
        # iterate through each file
        for file in os.listdir(path):
            # file name is in the form of "cat.0.jpg" which contains category
            # split by '.' to get 'cat' at index 0 in array
            category = file.split(".")[0]
            category = convert_into_label(category)
            img_array = cv2.imread(os.path.join(path,file)) #read image using opencv
            new_size = (128,128) #resize the image to (128,128,3)
            new_img_array = cv2.resize(img_array,new_size)
            training_images.append(new_img_array)
            training_labels.append(category)
    else:
        for file in os.listdir(path):
            id = file.split(".")[0]
            img_array = cv2.imread(os.path.join(path,file))
            new_size = (128,128)
            new_img_array = cv2.resize(img_array,new_size)
            testing_images.append(new_img_array)
            ids.append(id)
    end_time = time.time()
    length = end_time - start_time
    print(f"load_dataset in {mode}mode took {length:.3f}s")
    if "test" in mode:
        return np.array(testing_images),np.array(ids)
    return np.array(training_images), np.array(training_labels)

We’re having 25000 images of (128,128,3) shape. To train these many images on L-layer NN can take significant amount of time. Hence, we’ll make a small set out of this to train on a L-layer NN. For our thorough understanding on how backpropagation works, how to initialize parameters of L-layer Neural Network, we consider number of training examples to be 300 which is formed from a random shuffle of original train data (X_train and y_train).

Creation of new train datset with 300 training examples and 100 test examples from original train dataset (having 25000 images of (128,128,3) shape).

import pandas as pd
# setting up seed to make the shuffle same every time we run
np.random.seed(14)

# get random indices of length 300 which ranges from 1 to 25000
random_indices = np.random.randint(25000,size = 300)

# preparing new train dataset
X_train_new = [X_train[random_index] for random_index in random_indices]
y_train_new = [y_train[random_index] for random_index in random_indices]

# preparing new test dataset
# create new_test_dataset from X_train so that we know the labels, 
# make sure we don't include the indices in random_indices
available_indices = [index for index in range(25000) if index not in random_indices]
test_random_indices = np.random.randint(len(available_indices),size = 100)
X_test_new = [X_train[available_indices[index]] for index in test_random_indices]
y_test_new = [y_train[available_indices[index]] for index in test_random_indices]

Cats and Dogs Distribution of size 300 where we picked indices from training_images using NumPy. 0 represents “Cat” and 1 represent “Dog”

We’ve plotted the above figure to confirm that we’re not having an imbalanced training dataset. The below code describes the data profile we’re having.

# Explore your dataset 
m_train = X_train_new.shape[0] # number of training examples
num_px = X_train_new.shape[1] # number of features 
m_test = X_test_new.shape[0]  # number of test examples

print ("Number of training examples: " + str(m_train))
print ("Number of testing examples: " + str(m_test))
print ("Each image is of size: (" + str(num_px) + ", " + str(num_px) + ", 3)")
print ("X_train shape: " + str(X_train_new.shape))
print ("y_train shape: " + str(y_train_new.shape))
print ("X_test shape: " + str(X_test_new.shape))
print ("y_test shape: " + str(y_test_new.shape))

Data Profile of New Train Dataset

Feature Engineering Pipeline

Feature Engineering in Deep Neural Networks includes Flattening, Standardizing of Feature Values between 0 and 1 (to make sure Optimization Algorithms like Gradient Descent, Batch Gradient Descent to converge faster and minimize the loss)

# Reshape the training and test examples 

# made the x_train_flatten to be having a shape of (n_x,m)
# The "-1" makes reshape flatten the remaining dimensions
x_train_flatten = X_train_new.reshape(X_train_new.shape[0],-1).T 
x_test_flatten = X_test_new.reshape(X_test_new.shape[0], -1).T

# Standardize data to have feature values between 0 and 1.
x_train = x_train_flatten/255.
x_test = x_test_flatten/255.

print ("train_x's shape: " + str(x_train.shape))
print ("test_x's shape: " + str(x_test.shape))

Note that 1281283 = 49152. We’ve converted the train_shape from (300,128,128,3) to (49512, 300)

Defining the Neural Network Architecture

We define a deep neural network with number of layers and number of hidden units in each layer. For our understanding upon back propagation, optimisation and updating parameters, I’ve defined a 5 layer neural network defined as layer_dims = [train_x.shape[0], 64, 32, 16, 4, 1]. A Deep NN models works in the below way.

parameters =  initialize_parameters_deep (layers)
for iteration = 1 to number of iterations:
    perform forward propagation
    compute costs
    compute the grads using backpropagation
    update the grads using optimization algorithm
    print cost / loss for specific iteration (for epoch = 100 or so)
      append the costs for specific epoc for plotting loss / accuracy curves

parameters is a dictionary which consists of Weights & Biases for different levels. For example, parameters[“W1”] consists of weights for computation between layer 1 and input layer, parameter[“W2”] consists of weights for computation between layer 2 and layer 1, etc.

# parameters dict to store each weights and bias
# for example parameters[w1] gives a vector which has 
# weights between layer 1 and layer 0 and parameters[w2] 
# has weights between layer 2 and layer 1

# defining layers which consists of number of hidden neurons

layer_dims =[x_train.shape[0], 64, 32, 16,4,1]

def initialize_parameters_deep(layer_dims):
    """
    Initializes Weights as per He Initialization / Kaiming Initialization given by 
    W ~ N(0, 2/n^l) where n^l represents the number of neurons
    in current layer

    Arguements:
    layer_dims: A List:[int] which consists of number of neurons in each layer
    Returns
    parameters: A dictionary of parameters such as 
    "W1","b1","W2","b2","W3","b3",...."WL","bL"
     Wl has a shape of (layer_dims[l], layer_dims[l-1])
     bl has a shape of (layer_dims[l],1)
    """
    np.random.seed(1)
    parameters = {}
    L = len(layer_dims)
    for layer in range(1,L):
        parameters["W"+ str(layer)] = np.random.randn(layer_dims[layer],layer_dims[layer-1])*np.sqrt(2/layer_dims[layer-1])
        parameters["b"+str(layer)] = np.zeros((layer_dims[layer],1))
    return parameters
parameters = initialize_parameters_deep(layer_dims)

#iterating over keys and values of parameters
for key,value in parameters.items():
    print(f"{key} shape: {value.shape}",sep='\n')

shapes of different weights for different layers.

For deeper understanding of further process, I’ve taken a reference image from coursera deep learning specialization, which explains how forward propagation, backpropagation and optimization happens.

Image taken from Coursera Deep Learning Specialization

We’ve considered the above scenario, where we keep activation layer as ReLU from layer 1 to layer L-1, and layer L has Sigmoid activation which computes the probability of a picture being a cat or a dog. To perform forward propagation we define activation functions as below.

Forward Propagation: Neural Network

Below are the functions that are used in L_Layer_forward, which performs the forward propagation.

def sigmoid(Z):
    """
    Performs sigmoid function on a numpy array Z

    Arguments:

    Z: a numpy array

    Returns:

    A: Activation of Z which is sigmoid (Z) and cache (which can be used for backpropagation). Here cache is Z
    A,cache
    """
    A = np.exp(Z) / (1 + np.exp(Z))
    cache = Z
    return A, cache

def relu(Z):
    """
    Performs ReLU function on a numpy array Z

    Arguments:

    Z: a numpy array

    Returns:

    A: Activation of Z which is (Z) and cache (which can be used for backpropagation). Here cache is Z

    A,cache
    """
    A = np.maximum(0,Z)
    cache = Z
    return A, cache
def linear_forward(A,W,b):
    """
    Computes Linear Part of Forward Propagation

    Arguments:

    A: Activation of previous layer
    W: Weight of current layer
    b: bias of current layer

    Returns:
    Z -- the linear combination of input with weights
    cache - essentially A and b which can be later used for backpropagation
    """
    Z = W.dot(A) + b
    assert(Z.shape == (W.shape[0],A.shape[1]))
    cache = (A,W,b)
    return Z, cache

def linear_forward_activation(A_prev,W,b,activation):
    """
    Computes Linear + Activation Part of Forward Propagation

    Arguments:
    A: Activation of previous layer
    W: Weight of current layer
    b: bias of current layer
    activation: can be "relu" or "sigmoid"

    Returns:
    A -- Activation calculated for current layer using A_prev, W,b and activation
    cache - a tuple of (A_prev, W,b) which can be later used for backpropagation

    """
    if activation == "relu":
        Z,linear_cache = linear_forward(A_prev,W,b) # linear_cache contains A_prev, W, b
        A,activation_cache = relu(Z) # activation_cache contains Z
    else:
        Z,linear_cache = linear_forward(A_prev,W,b) # linear_cache contains A_prev, W, b
        A,activation_cache = sigmoid(Z) # activation_cache contains Z
    cache = (linear_cache,activation_cache) # ((A,W,b),Z)
    # pack linear_cache, activation_cache so that we have every variable that can be used in backpropagation
    return A, cache

# writing L - Layer Forward function

def L_Layer_forward(X,parameters):
    """
    Computes Linear - ReLU Activation Propagation for L-1 layers and Linear - Sigmoid for Lth Layer 

    Arguments:

    X: Input Data (assumed as layer 0)
    parameters: weights and biases dictionary

    Returns:
    AL -- Activation at Lth Layer
    caches -- A list of (linear_cache,activation_cache) at each layer.
    """
    caches = []
    A = X
    # number of layers
    L = len(parameters)//2
    for layer in range(1,L):
        # activation is relu
        A,cache = linear_forward_activation(A,parameters["W"+str(layer)],parameters["b"+str(layer)],activation="relu")
        caches.append(cache)
    # last layer activation is relu
    AL,cache = linear_forward_activation(A,parameters["W"+str(L)],parameters["b"+str(L)],activation="sigmoid")
    caches.append(cache)
    return AL, caches
AL,caches = L_Layer_forward(X_train_new,parameters)

Computing Cost

Cost function for binary classification is the average of loss functions over the entire set. Loss function is Binary Cross Entropy for binary classifications which is defined below.

Cost computation without regularization

# compute binary cross entropy loss between AL and Y

def compute_cost(AL,y):
    """
    Computes Binary Cross Entropy Loss between AL and y

    Arguments:
    AL: Activation of Lth Layer
    y: true label

    Return:
    Binary Cross Entropy Cost
    """
    try:
        assert(AL.shape == y.shape)
    except AssertionError:
        y = y.reshape(AL.shape[0],AL.shape[1])
    # number of examples
    m = y.shape[1]
    # computing loss
    log_probs = y*np.log(AL) + (1-y)*np.log(1-AL)
    cost = -np.sum(log_probs,keepdims=True)/m
    cost = np.squeeze(cost)
    assert(cost.shape == ())
    return cost

compute_cost(AL,y_train_new)

Back propagation

Backpropagation is one of the crucial steps to understand how gradients are computed. It calculates gradients based on differentiation of costs with respect to parameters.

Back propagation: Derivation using Chain Rule

def sigmoid_backward(dA,activation_cache):
    """
    calculates dZ, using dA and activation_cache. 
    Here activation_cache is equivalent to cache given by sigmoid(Z) function, which is indeed Z
    activation_cache = Z.
    Arguements:
    dA: Activation of layer l
    activation_cache: equals to Z
    Returns:
    dZ: gradient of Z, which can later be used in calculation of dW, db
    """
    Z = activation_cache
    # dZ = dA * g'(Z) (element wise multiplication)
    # here g'(Z) = g(Z) * (1-g(Z))
    g_z = np.exp(Z)/ (1 + np.exp(Z)) # sigmoid function
    d_gz = g_z * (1 - g_z) # d_gz = g'(z) = g(z) * (1-g(z))
    dZ = dA * d_gz
    assert(dZ.shape == Z.shape)
    return dZ
def relu_backward(dA,activation_cache):
    """
    calculates dZ, using dA and activation_cache. 
    Here activation_cache is equivalent to cache given by relu(Z) function, which is indeed Z
    activation_cache = Z.
    Arguements:
    dA: Activation of layer l
    activation_cache: equals to Z
    Returns:
    dZ: gradient of Z, which can later be used in calculation of dW, db
    """
    Z = activation_cache
    # dZ = dA * g'(Z)
    # here g'(Z) = 1 for z > 0 and g'(Z) = 0 for z <= 0
    # dZ = dA for Z > 0
    dZ = np.copy(dA)
    dZ = np.where(Z <= 0, 0,dZ)
    assert(dZ.shape == Z.shape)
    return dZ

# using dZ to calculate dW, db and dA^{[l-1]}

def linear_backward(dZ,linear_cache):
    """
    Implement backpropagation for a single layer l
    Arguments:
    dZ: derivative of cost with respect to Z (dJ/dZ)
    linear_cache: consists of tuple of (A_prev,W,b). this is the output of linear_forward function which gives Z and (A_prev,W,b)
    Returns:
    dA_prev: derivative of cost with respect to A_prev, same shape as A_prev
    dW: derivative of cost with respect to W, same shape as W
    db: derivative of cost with respect to b, same shape as b
    """
    A_prev,W,b = linear_cache
    # A_prev is of shape (layer[l-1],m), W is of shape (layer[l], layer[l-1]), b is of shape (layer[l],1)
    # dZ is of same shape as Z, which is (layer[l],m)
    # number of examples
    m = A_prev.shape[1]
    # calculation of dA_prev, dW, db
    dA_prev = np.dot(W.T,dZ)
    dW = np.dot(dZ,A_prev.T)*(1./m)
    db = np.sum(dZ,axis = 1,keepdims=True)*(1./m) # perform summation along the row, i.e, consider a single feature addition among all examples
    assert (dA_prev.shape == A_prev.shape)
    assert (dW.shape == W.shape)
    assert (db.shape == b.shape)
    return dA_prev, dW, db

def linear_activation_backward(dA,cache,activation):
    """
    Implement backpropagation for a linear->activation layer
    Arguments:
    dZ: derivative of cost with respect to Z (dJ/dZ), same shape as Z
    cache: consists of tuple of (linear_cache,activation_cache). this is the output of linear_forward function which gives Z and (A_prev,W,b)
    Returns:
    dA_prev: derivative of cost with respect to A_prev, same shape as A_prev
    dW: derivative of cost with respect to W, same shape as W
    db: derivative of cost with respect to b, same shape as b
    """
    linear_cache, activation_cache = cache
    if activation=='relu':
        #compute dZ from relu_backward
        dZ = relu_backward(dA,activation_cache)
    elif activation == 'sigmoid':
        dZ = sigmoid_backward(dA,activation_cache)
    else:
        print(f"Please enter correct activation parameter!, you have entered {activation}, which has to be either relu or sigmoid")
    # compute dA_prev, dW, db from dZ using linear_backward
    dA_prev, dW, db = linear_backward(dZ, linear_cache)
    return dA_prev, dW, db

# using the above functions, we iterate through each layer and calculate gradients dW, db through L_model_backward function

def L_model_backward(AL,y,caches):
    """
    Implements backpropagation for given Neural Network

    Arguments:
    AL: Activation for last layer
    Y: true labels equivalent to y_train
    caches: A list of (linear_cache,activation_cache) for each layer

    Returns:
    grads: A dictionary which consists of dA,dW, db for each layer, i.e., grads["W1"] consists of gradient for W1
    """
    grads = {}
    L = len(caches)
    m = AL.shape[1]
    # initialize backpropagation
    # compute dAL for the last layer
    assert(AL.shape == y.shape)
    dAL = -(np.divide(y,AL) - np.divide(1-y,(1-AL)))
    # compute dZL for the last layer
    cache_L = caches[L-1]
    dA_prev,dW,db = linear_activation_backward(dAL,cache_L,activation="sigmoid")
    grads["dA"+str(L-1)],grads["dW"+str(L)],grads["db"+str(L)] = dA_prev,dW,db
    # compute grads for penultimate layer to the first layer
    for l in reversed(range(L-1)):
        # lth layer contains relu activation function
        grads["dA"+str(l)],grads["dW"+str(l+1)],grads["db"+str(l+1)] = linear_activation_backward(grads["dA"+str(l+1)],caches[l],activation="relu")
    return grads

Updating Parameters

Parameters are updated through gradient descent algorithm. Cost is minimized through Gradient Descent Algorithm by using gradients calculated from back propagation.

Gradient Descent Algorithm

def update_parameters(parameters,grads,learning_rate):
    """
    Updates Weights by Gradient Descent Algorithm;
    Arguments:
    parameters: a dictionary of weights and biases
    grads: a dictionary of grads of weights and biases
    learning_rate: a parameter that can be used to update the 
    returns:
    parameters: a dictionary of weights and biases with updated parameters
    """
    L = len(parameters)//2
    for l in range(1,L+1):
        parameters["W" + str(l)] -= learning_rate*grads["dW"+str(l)]
        parameters["b" + str(l)] -= learning_rate*grads["db"+str(l)]
    return parameters

Training

Model function can be used to sequentially call the above described functions to train the data. For every 100 epochs trained, we print the cost to check cost’s decrease.

def L_layer_model(X,y,layer_dims, learning_rate = 0.01, num_iterations = 5000, print_cost = False):
    """
    Implements a L-layer neural network: [LINEAR->RELU]*(L-1)->LINEAR->SIGMOID.

    Arguments:
    X -- input data, of shape (n_x, number of examples)
    Y -- true "label" vector (containing 0 if cat, 1 if dog), of shape (1, number of examples)
    layers_dims -- list containing the input size and each layer size, of length (number of layers + 1).
    learning_rate -- learning rate of the gradient descent update rule
    num_iterations -- number of iterations of the optimization loop
    print_cost -- if True, it prints the cost every 100 steps

    Returns:
    parameters -- parameters learnt by the model. They can then be used to predict.
    """
    # reshape y if not in the format of (1, number of examples)
    try:
        assert(y.shape == (1,X.shape[1]))
    except AssertionError:
        y = y.reshape(1,X.shape[1])
    # keeping random seed
    np.random.seed(14)
    costs = []
    # initialize parameters
    parameters = initialize_parameters_deep(layer_dims)
    for epoch in range(num_iterations):
        # perform forward propagation 
        AL,caches = L_Layer_forward(X,parameters)
        # compute cost
        cost = compute_cost(AL,y)
        # perform backpropagation
        grads = L_model_backward(AL,y,caches)
        # perform optimization
        parameters = update_parameters(parameters,grads,learning_rate=learning_rate)
        # print costs
        if epoch%100 == 0 or epoch == num_iterations - 1:
            print(f"Cost after iteration {epoch}: {np.squeeze(cost)}")
        if epoch%100 == 0:
            costs.append(np.squeeze(cost))
    return parameters,costs
parameters, costs = L_layer_model(
X_train_new,y_train_new,layer_dims, num_iterations = 5000, print_cost = True)

The above function performs training by forward propagation, cost computation, back propagation, optimization and printing the cost. The costs are clearly decreased when we plot a Training Loss Curve as shown below.

Predictions

Predictions are used to assess the model’s performance against the training and test datasets. It uses tuned parameters obtained from L_layer_model function to predict the unknown data and print the training and test accuracy.

def predict(X,y,parameters):
    """
    This function is used to predict the results of a  L-layer neural network. Along with predictions, it prints accuracy

    Arguments:
    X -- data set of examples you would like to label
    parameters -- parameters of the trained model

    Returns:
    p -- predictions for the given dataset X
    """
    L = len(parameters)//2
    probs,caches = L_Layer_forward(X,parameters)
    try:
        assert (probs.shape == y.shape)
    except AssertionError:
        y = y.reshape(probs.shape[0],probs.shape[1])
    m = probs.shape[1]
    predictions = np.where(probs > 0.5,1,0)
    print("Accuracy: " + str(np.sum(predictions == y)/m))
    return predictions

Ahh!, the model is poorly performing on the test dataset and performs highly on train dataset. The model is said to be overfitting as it can’t able to generalize on new data. We can improve the performance of this model by using PyTorch and Convolutional Neural Networks which I’ll explain in the upcoming episode S1 E2.