Naive Bayes Classifier Explained: Assumptions, Types, and Uses

What is Naive Bayes Classifier?

Naive Bayes classifier is a type of probabilistic classifier that makes predictions based on Bayes’ theorem with the “naive” assumption of feature independence. It’s called “naive” because it assumes that the presence of a particular feature in a class is independent of the presence of other features, given the class label. Despite this simplifying assumption, Naive Bayes classifiers are powerful and widely used in various machine learning tasks, particularly in text classification and sentiment analysis.

The Naive Bayes classifier calculates the probability of a data point belonging to each class based on the observed features. It then predicts the class with the highest probability as the final prediction.

Why it is Called Naive Bayes?

The term “Naive Bayes” in machine learning can be broken down into two parts:

• Naive: Naive Bayes classifiers make a simplifying assumption — they treat all features (characteristics used for classification) as independent of each other, given the class label. This means the presence or absence of one feature doesn’t influence the presence or absence of another feature, considering the class is already known. In reality, this assumption isn’t always true.
• Bayes: This refers to Bayes’ theorem, a theorem in probability theory that helps to calculate the conditional probability of an event (the class label) happening, given another event (the features) has already occurred.

Why Naive? This assumption of independence is often unrealistic in real-world data. For instance, in spam filtering, the presence of the word “free” might make the presence of “discount” more likely. However, despite this simplification, Naive Bayes often performs well — that’s why it’s considered “naive” because it makes an assumption that might not always hold true.

“Naive Bayes” essentially means a classification algorithm that uses Bayes’ theorem but makes a simplifying assumption about the independence of features.

Bayes’ Theorem

Source

You might be aware of that Bayes’ Theorem is a fundamental principle in probability theory that provides a way to update our beliefs or the probability of an event based on new evidence or information.

It’s named after the Reverend Thomas Bayes, an 18th-century statistician and theologian who made significant contributions to probability theory. Furthermore, the theorem has applications in various fields, including statistics, machine learning, science, and even philosophy.

Bayes’ Theorem helps us calculate conditional probabilities. Here, Conditional probability is the probability of an event occurring given that another event has already occurred. We write Bayes’ Theorem in mathematics like this:

where,

• P(A∣B): conditional probability of event A occurring given that event B has occurred.
• P(B∣A): the probability of event B occurring given that event A has already occurred.
• P(A): prior probability of event A, before considering any new evidence.
• P(B): probability of event B.

Math behind Naive Bayes Algorithm

Given a features vector X=(x1,x2,…,xn) and a class variable y, Bayes Theorem states that:

We’re interested in calculating the posterior probability P(y | X) from the likelihood P(X | y) and prior probabilities P(y),P(X).

Using the chain rule, the likelihood P(X ∣ y) can be decomposed as:

but because of the Naive’s conditional independence assumption, the conditional probabilities are independent of each other.

Thus, by conditional independence, we have:

And as denominator remains constant for all values, the posterior probability can then be:

The Naive Bayes classifier combines this model with a decision rule. One common rule is to pick the hypothesis that’s most probable; this is known as the maximum a posteriori or MAP decision rule.

Assumption of Naive Bayes

The fundamental Naive Bayes assumption is that each feature makes an:

• Feature independence: The features of the data are conditionally independent of each other, given the class label.
• Continuous features are normally distributed: If a feature is continuous, then it is assumed to be normally distributed within each class.
• Discrete features have multinomial distributions: If a feature is discrete, then it is assumed to have a multinomial distribution within each class.
• Features are equally important: All features are assumed to contribute equally to the prediction of the class label.
• No missing data: The data should not contain any missing values.

Flavors of Naive Bayes Model

1. Gaussian Naive Bayes (Gaussian NB):

• Works well with continuous features.
• Handles data where features can take on any numerical value within a range, like temperature, height, weight, or sensor readings.

Assumption: Assumes features follow a normal distribution (also known as a Gaussian distribution). This is a bell-shaped curve where most data points cluster around the average value.

Source

Applications:

Classifying handwritten digits based on their pixel intensities (continuous values).

Predicting stock prices based on historical trends.

Anomaly detection in sensor data (identifying unusual sensor readings).

The decision rule for Gaussian Naive Bayes is:

Where *μ is the mean and σ *is the standard deviation that we have to estimate from the data.

2. Multinomial Naive Bayes (Multinomial NB):

• Multinomial Naive Bayes is the most common type and is particularly effective for text classification.
• Handles discrete features, meaning features represent counts or frequencies. These are often integer values but can also be categorical data represented numerically (e.g., word counts in a document, number of times a specific product is purchased by a customer).

Assumption: Assumes features follow a multinomial distribution. This means the probabilities of each feature value (e.g., word frequency) sum up to 1.

Applications:

Classifying emails as spam or not spam based on word frequency.

Sentiment analysis classifying reviews as positive, negative, or neutral.

Document classification categorizing news articles by topic (e.g., sports, politics, business).

The decision rule for Multinominal Naive Bayes is:

where , x represents the count of the feature (like a specific word) in samples belonging to class y.

3. Bernoulli Naive Bayes (Bernoulli NB):

• Designed for binary features.
• Handles data where each feature has only two possible values, typically represented as 0 or 1. Often used for presence/absence data.

Assumption: Each feature is independent and can be modeled by a Bernoulli distribution. This means the probability of each feature value (0 or 1) is independent of other features.

Applications:

Sentiment analysis classifying tweets as positive or negative based on the presence of specific words (e.g., “happy” = 1, “sad” = 0).

Image classification identifying the presence or absence of objects in an image (e.g., “car” = 1, “not car” = 0).

Gene expression analysis classifying genes as “on” or “off” based on their activity level.

The decision rule for Bernoulli Naive Bayes is:

where, i is an event, and x_i needs to be binary (0 or 1).

How Naive Bays really works:

Let’s explain it using an example to make things clear:

Assume we have a bunch of emails that we want to classify as spam or not spam.

Our dataset has 15 Not Spam emails and 10 Spam emails. Some analysis had been done, and the frequency of each word had been recorded as shown below:

Note: Stop Words like “the”, “a”, “on”, “is”, “all” had been removed as they do not carry important meaning and are usually removed from texts. The same thing applies to numbers and punctuations.

exploring some probabilities:

• P(Dear|Not Spam) = 8/34
• P(Visit|Not Spam) = 2/34
• P(Dear|Spam) = 3/47
• P(Visit|Spam) = 6/47

and so on.

now assume we have the message “Hello friend” and we want to know whether it is a spam or not.

so, using Bayes’ Theorem

ignoring the denominator

But, P(Hello friend | Not Spam) = 0, as this case (Hello friend) doesn’t exist in our dataset, i.e. we deal with single words, not the whole sentence, and the same for P(Hello friend | Spam) will be zero as well, which in turn will make both probabilities of being a spam and not spam both are zero, which has no meaning!!

But wait!! we said that the Naive Bayes assumes that the features we use to predict the target are independent.

so,

now let’s calculate the probability of being spam using the same procedure:

so, the message “Hello friend” is not a spam.

now, let’s try another example:

assume the message “dear visit dinner money money money”. It’s obvious that it’s a spam, but let’s see what Naive Bayes will say.

oops!! Naive Bays says that this message is not a spam?!!!

This happened because the word “dinner” does not appear in the spam dataset, so that P(dinner | spam) = 0, hence all other probabilities will have no effect.

This is called the Zero-Frequency Problem.

and to solve that we can use Laplace smoothing.

What is Laplace smoothing?

Are you thinking “such a complicated term this is!!!!! LAPLACE SMOOTHING!!” Well, it’s not so. It’s a pretty simple and problem-solving concept in the case of zero probability. Have a read of this section, you will get to know how important this part is in the Naive Bayes implementation.

Laplace smoothing is used to solve the problem of zero probability in Naive Bayes. We add a smoothing value ⍺ to each of the counts so that the probability of an attribute, given any class, never becomes zero. The probability is then calculated as:

where k represents the number of dimensions (features) in the data. It will be non-zero and have reasonable probability, and the problem of zero probability will be resolved.

Here, alpha represents the smoothing parameter,

If we choose a value of alpha!=0 (not equal to 0), the probability will no longer be zero even if a word is not present in the training dataset.

What is Logarithmic Probability?

Amid all these formulae, there is one thing of particular interest here. Say we have a large number of input features, so to calculate the final value we notice that after multiplying all the conditional probabilities (0<P<1), our result obtained is rather a small value approaching 0.

Now, working with such small values can be difficult as it can lead to underflow of numeric precision and our algorithm may approximate the calculated value to 0! Hence, this is when the concept of logarithmic probability comes to our rescue where we take the log of the obtained value to avoid this problem.

This transformation works (and is rather elegant) because:

• A probability is always between 0 and 1 so a log probability is always between −∞ and 0 (a much bigger range of numbers than between 0 and 1).

• If a < b, then ln a < ln b.

This property of logarithms means that comparing probabilities to figure out which one is the biggest is equivalent to comparing the log probabilities.

• Also the math part of it works out rather well because the logarithm of a product is the sum of the individual log probabilities.

So, we can definitely conclude that log probabilities solve the precision problem that we may encounter during the normal calculations.

What is Hyperparameter tuning?

Naive Bayes is a classification machine learning algorithm. Following are the graphs of the dataset showing their classification as per model gets trained.

In the previous section, where we learned Laplace smoothing, as we increase ⍺ also called a hyperparameter, the likelihood of p(x_i|Yes) and p(x_i|No) tends to 0.5. They become biased towards the value of 0.5. In this case, the problem of underfitting occurs.

Similarly, when the value of ⍺ is very small, it will give importance to even very rare words or data points from the training data, in this case, an overfitting problem will occur. Hence, for the perfect fit of the value, we tune the model. For cross-checking, we check the performance for different hyperparameters.

Applications of Naive Bayes

Naive Bayes has been successfully applied in numerous fields, underscoring its versatility:

Source

• Spam Detection: One of the earliest and most famous applications of Naive Bayes is in the filtering of unwanted emails based on the likelihood of certain words appearing in spam versus non-spam emails.
• Sentiment Analysis: Naive Bayes is commonly used in sentiment analysis, determining whether a text expresses positive, negative, or neutral sentiments, particularly useful in social media monitoring and market research.
• Document Classification: It is extensively used in classifying documents, such as categorizing news articles into various topics or organizing books into genres.
• Healthcare: Naive Bayes has applications in the medical field for disease prediction and discovering relationships between various risk factors and diagnosis.

Strengths and Weaknesses of Naive Bayes

Advantages of Using Naive Bayes

Naive Bayes classifier holds several advantages that make it a valuable tool in a data scientist’s arsenal:

• Efficiency: Naive Bayes is known for its simplicity and speed. It can make quick predictions even with large datasets, which is invaluable in real-time applications.
• Easy to Implement: With fewer parameters to tune, Naive Bayes can be easier to implement compared to more complex models like neural networks.
• Good Performance with Small Data: Unlike some models that require vast amounts of training data to perform well, Naive Bayes can achieve good results even with a smaller dataset.
• Probabilistic Interpretation: The model provides probabilities for outcomes, offering more insight into the results, such as how likely a given class is the correct classification.

Limitations and Considerations

Despite its strengths, Naive Bayes comes with limitations that might affect its performance:

• Independence Assumption: The biggest limitation is the assumption of independent predictors. In real-world scenarios, features often influence each other, and this assumption can lead to incorrect predictions.
• Zero-Frequency Problem: If a categorical variable has a category in test data set which was not observed in training data set, it will assign a 0 probability and will be unable to make a prediction. This is often mitigated by a smoothing technique.
• Biased Estimates: Because it relies heavily on the actual distribution of classes and features in the training set, Naive Bayes can produce biased estimates if the training data is not representative.

Conclusion

The Naive Bayes algorithm is a powerful and straightforward algorithm that excels in various classification tasks, particularly when dealing with large datasets and high-dimensional data. Its simplicity and speed make it a go-to choice for many practical applications, including spam filtering, sentiment analysis, and medical diagnosis. Despite its assumption of feature independence, which may not always hold true in real-world scenarios, Naive Bayes often performs surprisingly well, thanks to the robustness of probabilistic reasoning.

Moreover, Naive Bayes is not only easy to implement but also interpretable, providing clear insights into the decision-making process. By understanding the underlying probabilities, data scientists can gain valuable intuition about the relationships within their data. As with any machine learning algorithm, it’s important to evaluate its performance in the context of your specific problem and consider combining it with other techniques when appropriate.