Neural Network

A recap of Andrew Ng course (Machine Learning Specialization) on Coursera

Course 2 Week 1

Photo by Jezael Melgoza on Unsplash

Neural Network

With the start of internet and mobile phone in the past few decades, the amount of data we could feed to the learning algorithm has increased a lot. However, the traditional AI like logistic regression wouldn’t be able to improve its performance even with this amount of data. On the contrary, larger and more complex learning algorithm benefits from more data and faster computing powers.

Demand Prediction: try to predict whether a product will be a top-seller

This analysis is used by retailers today to plan better inventory level and marketing campaigns. we can think of a neuron is doing the sigmoid function below, the neuron takes price as input and output the probability of being top seller. In deep learning, we use a different word to refer to the sigmoid function, we call it activation function. activation is a neuroscience word, and it stands for how much a neuron is sending a high output to other neurons downstream from it.

Building a neural network is actually taking a bunch of this neuron and wire them together. Wen we were to predict the probability of being a top seller, we naturally think of 3 factors: affordability, buyer awareness, and perceived quality. In the graph below, we add 3 more features to the input, and each of the feature is related to parts of the 3 factors. For example, we would think that affordability function takes price and shipping fee as inputs; and perceived awareness is related to marketing..etc. The aggregated result of the 3 factors will be the final probability of being top seller.

In deep learning, we would deem those 3 factors as neurons which together form a layer. A layer can have multiple neurons. For the final layer that shows the probability, we call that an output layer.

In addition, we call those neurons in the first layer activations, which is a biological term that refers to the degree that a neuron is sending high output value or sending many electrical impulses to other neurons, to the downstream from it (the neuron in the output layer.) Namely, we can think of the three factors: affordability, awareness and quality are the activations for the three neurons in the first layer respectively; while the probability is the activation for the neuron in the output layer.

To put it even more simply, this neural network takes 4 numbers, runs through the first layer which gets 3 activation values, and then finally outputs 1 number. In the above example, the three activation functions take the inputs that we manually chose; however, this would be too many works for a real-world neural network. In a real-world neural network, the middle layer actually takes every single feature from the input layer as inputs, and we can imagine that through the process of learning, the affordability activation function will learn to focus more on price and shipping fee.

An easy way to think of deep learning is that it is still a logistic regression, which simply takes 3 input features: affordability, awareness and quality. The difference is that deep learning is not taking from the original raw input, instead it transformed the raw inputs (price, shipping fee, marketing and material) into more complex features (affordability, awareness and quality) that are more predictive than the original ones.

We can also say that deep learning model learns from the raw input and create customized features. In contrast to deep learning, when doing traditional ML, we need to conduct feature engineering to create new features from the raw features. For example, we can multiply two features to create a third new feature. The ability to learn its own feature is what makes deep learning so powerful.

Here are some more examples of neural network architectures. Some people also call them multilayer perceptron.

Picking the proper number of layers and the number of neurons within each layer will be covered in the later blog posts.

Face Recognition

this time we will look at neural network which takes as input a vector with 1 million features and then output the identity of the person from the input. As shown below, we have multiple hidden layers. If we look at the first hidden layer, we will notice that each of the neuron in it is looking at presence or absence of very short edges in the image from different angles.

And then the second hidden layer learns to group the outputs from the first layers and starts to recognize parts of the faces. It is learning the presence or absence of those parts, such as eyes, ears.

The first layer focuses on smaller regions in the image, and the second focuses on middle size regions, and the third layer focuses on largest regions.