Naive Bayes Algorithm

Naive bayes is a probability based machine learning algorithm which is based on bayes theorem. The Niave Bayes classifier is a common supervised machine leaning algorithm popularly used for text classification.

In every algorithm we make some assumptions like in KNN we assume that the neighborhood datapoints belong to the same class. Likewise in Naive Bayes we assume that features are conditionally independent. That is the occurrence of one feature does not affect the occurrence of another feature.

Bayes theorem is a common terminology that we would have come across in our probability classes in our high school. Before we understand Naive bayes theorem we should first understand what bayes algorithm is.

Probability of A given B is understood as — what is the probability of A given the fact that B has already occurred. It is nothing but finding the probability of something conditioned on the fact that something else has already happened.

In the above formula,

P(A/B) → Posterior probability

P(B/A) → Likelihood

P(A) → Prior

P(B) → Evidence

Now lets consider that a person plays 8 out of 10 days. So Probability(play) = 8/10. Probability(not play) = 2/10. Now if we want to know P(play/rain), then it becomes conditional probability. That is, what is the probability of the person play given the condition that it rains. Now on applying bayes theorem,

P(play/rain) = P(rain/play) * P(play) / P(rain)

Niave Bayes with 1 conditional variable

Now consider the above table. If we want to find the probability of the person playing golf or not if it is raining then I need to compute P(outlook = rainy| Play Golf = No) and P(outlook = rainy| Play Golf = Yes). Which ever probability is greater we can conclude the result.

P(Outlook = Rainy ) = 5/14
P(Play Golf = Yes | Outlook = Rainy) = 2/9
P(Play Golf = No | Outlook = Rainy) = 3/5
P(Play Golf = Yes) = 9/14
P(Play Golf = No) = 5/14

P(Play Golf=Yes | Outlook=Rainy)
= P(Outlook=Rainy|Play Golf=Yes)*P(Play Golf = Yes)/P(Outlook=Rainy)
= 2/9*9/14*14/5=2/5 = 0.4

P(Play Golf=No|Outlook=Rainy)
= P(Outlook=Rainy|Play Golf=No)*P(Play Golf = No)/P(Outlook=Rainy)
= 3/5*5/14*14/5=3/5 = 0.6

Now we arrive at a probability where P(play Golf = No| Outlook = Rainy) is greater. So we can conclude that the person would not play golf when it is raining.

Naive Bayes with more than 1 conditional variables

In the previous example we just looked at one conditional variable. Now let us consider more than one conditional variable. Now we have to find if a person plays golf or not based on outlook is rainy , temperature is cool, humidity is high and windy is true.

According to bayes theorem, P(y/X) is given as

Here variable y is the class lable that is play golf or not. Variable X represents the parameters/features.

X is given as,

Here each x1, x2,….xn represents each separate feature like outlook, temperature, humidity and windy in our case.

Now, on expanding using chain rule we get,

Now, you can obtain the values for each by looking at the dataset and substitute them into the equation. For all entries in the dataset, the denominator does not change, it remain static. Therefore, the denominator can be removed and a proportionality can be introduced.

In our case we have only two class outcomes that is yes or no. There can be cases when the classes could be multi variate. There we will find the class y which has the maximum probability.

For our example where we need to find if the person paly golf or not given

Outlook = rainy , Temperature =cool, Humidity = high, Windy = true

For this we first need to construct a frequency table and likelihood table which looks like the below image.

Now based on these values I can calculate the my posterior probability.

What is Laplace smoothing?

While calculating the above probability if there is no value for humidity high the P(humidity/yes) is going to be 0. So while calculating P(yes/X), the value would be 0 as P(humidity/yes) is 0. The probability estimate becoming 0 is extremely dangerous. So we need to look into some better way of handling this.

Laplace smoothing or additive smoothing is a technique used in naive bayes algorithm to over the problem of zero probabilities which can help the model generalize better on unseen data.

The method is basically adding alpha to the numerator and and adding alpha*k to the denominator. ‘k’ is basically the number of class outcomes, in out case it is 2. Alpha is generally 1, basically a thumb rule. Lets understand the interpretation oof changing alpha going further.

Consider I’m doing a sentiment analysis problem. Using Laplace smoothing, we can represent P(w’|positive) as

Here, alpha represents the smoothing parameter, K represents the number of class outcomes, and N represents the number of reviews with y=positive

If we choose a value of alpha!=0 (not equal to 0), the probability will no longer be zero even if a word is not present in the training dataset.

Interpretation of changing alpha

Let’s say the occurrence of word w is 3 with y=positive in training data. Assuming we have 2 features in our dataset, i.e., K=2 and N=100 (total number of positive reviews).

Case 1- when alpha=1

P(w’|positive) = 3/102

Case 2- when alpha = 100

P(w’|positive) = 103/300

Case 3- when alpha=1000

P(w’|positive) = 1003/2100

As alpha increases, the likelihood probability moves towards uniform distribution (0.5). Most of the time, alpha = 1 is being used to remove the problem of zero probability.

Types of Naïve Bayes Model

1. Gaussian Naïve Bayes

In gaussian naïve bayes the continuous values associated with each feature is assumed to follow gaussian distribution. A random variable is said to follow a gaussian/normal distribution when plotted gives a bell shaped curve which is symmetric about the mean.

The likelihood of the feature is assumed to be gaussian and hence the conditional probability is given by:

2. Multinomial Naïve Bayes

Multinomial naïve bayes is mostly used in document classification problem like email spam classification, sentiment analysis etc. Multinomial naïve bayes is a variant mainly used for handling discrete data i.e., mainly like text classification problem where the features are word counts or term frequencies.

3. Bernoulli Naïve Bayes

Bernoulli naïve bayes works very similar to multinomial naïve bayes except the difference that the predictor variables are independent Boolean variable like if a particular word is present in a document or not represented by either 1 or 0. This model is often used for document classification task.

Advantages of Naïve Bayes

1. It is a very simple and fast algorithm.
2. Naive Bayes is very popularly used in text classification problem (due to its better result in multiclass problems and independence rule) have high success rate compared to other algos. It is mainly used in use cases like spam classification, text classification and sentiment analysis problems.
3. It is also easily interpretable as we are already making use of likelihood.
4. The outliers i.e., a word occurs in test data but not present in the vocabulary — are handled by Laplace smoothing.
5. Naïve bayes can handle categorical, binary and numerical features although it is popularly used for text data.
6. Can be used for both binary and multi-class classification.
7. The runtime space complexity is also very less as the algorithm has to store only the likelihood values and the prior probability.

Disadvantages of Naïve Bayes

1. Naïve bayes assumes conditional independence of features. But in real life the features are not always independent. For example in a hotel review calcification problem, for a positive review if I encounter a word amazing then there is high chance that I will encounter words like good, excellent, best etc.
2. Laplace smoothing is a must otherwise would end up with probability values to be 0 i.e., overfitting.

Refrences

1. https://www.analyticsvidhya.com/blog/2017/09/naive-bayes-explained/

2. https://www.analyticsvidhya.com/blog/2017/09/naive-bayes-explained/