Complete Sentiment Analysis Of News Data | Learning by Doing!

Ever feel like your day starts on a sour note after reading the latest headlines? If you’re tired of the endless stream of negative news, this project will give you the tools to categorize news stories into “good” or “bad” categories. The goal is to use machine learning to automatically classify news and help you curate a more positive start to your day! and well get some hands on experience in key Machine Learning concepts and algorithms.

Cover Image

Let’s dive into how we can achieve this using Python and some of the most common machine learning techniques. We’ll explore important concepts like TF-IDF vectorization, Grid Search, and a variety of algorithms that will help classify the news for you.

I used a crawler to extract text data from two Nepali news sites for the dataset. You can learn how to create a crawler by checking out the guide here.

Step 1: Importing Our Trusty Tools

Before we begin, we need our weapons. We’re pulling out a whole arsenal of tools like pandas, seaborn, scikit-learn, and a few others. These bad boys will help us crunch data, create models, and look smart while doing it.

import time
import pandas as pd
import seaborn as sns
import re
import nltk
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.metrics import accuracy_score, confusion_matrix, precision_score, recall_score
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
import matplotlib.pyplot as plt
import pickle

Woah! That’s a lot of imports but they are very very important !

Imagine trying to make pizza without dough, sauce, cheese, and a ridiculously large oven. That’s what coding without libraries is like.

Step 2: Data Analysis Time!

Alright, let’s take a peek at our dataset to see what kind of news stories we’re working with. We’ll check for missing values, duplicate rows, and make sure everything is in tip-top shape and also create a count plot to visualize the data.

def getDataAnalysis(trainingData):
    # simple data analysis , since this is for text based analysis there is not much analysis to do.
    # Lets do some basic

    #Total Data
    print(f'Total Data:\n,{trainingData.shape}')
    # Checking the columns
    print(f'Columns are : \n {trainingData.columns}')

    # Checking the data type
    print(f'Data Type : \n  {trainingData.dtypes}')

    # Checking for null values
    print(f'Null Values : \n {trainingData.isnull().sum()}')

    # # Checking for duplicates comment  this out when needed
    # print(f'Duplicates : \n {trainingData.duplicated().sum()}')
    #
    # # There is one duplicate lets remove
    # trainingData.drop_duplicates(inplace=True)
    #
    # # Recheck for duplicates again
    # print(f'Duplicates : \n {trainingData.duplicated().sum()}')

    # Lets do simple count plot
    # Using palette to add custom colors
    sns.countplot(data=trainingData, x='label', palette=['#A30000', "#4CB140"])
    plt.title("Count Plot Of Good News and Bad News")
    # using and creating patches to add custom label
    red_patch = mpatches.Patch(color='#A30000', label='0 Bad News')
    blue_patch = mpatches.Patch(color='#4CB140', label='1 Good News')
    plt.legend(handles=[red_patch, blue_patch])
    plt.show()
    # The count of bad news is more

Count Plot

After running this, we find out bad news dominates — yikes! 😬 But don’t worry, our machine learning models will figure out a way to differentiate the doom from the sunshine.

Step 3: Cleaning up the Mess 🧹

News data can be messy — think URLs, random symbols, and all kinds of digital clutter. Time for a quick spring cleaning!

def processCleanData(text):
    # changing to lower case
    text = text.lower()
    # removing symbols
    text = re.sub(r'@\S+', '', text)
    # removing links
    text = re.sub(r'http\S+', '', text)
    # removing pictures
    text = re.sub(r'.pic\S+', '', text)
    # removing other characters excpet text
    text = re.sub(r'[^a-zA-Z+]', ' ', text)
    # removing punctuation
    # getting punctuation list of string.punctation
    text = "".join([char for char in text if char not in string.punctuation])
    # tokenizing the workds
    words = nltk.word_tokenize(text)
    # using Lancaster stemmer to step the words
    # example eating eater becones eat
    words = list(map(lambda x: stemmer.stem(x), words))
    # removing and joining the stop words
    text = " ".join([char for char in words if char not in stopwords and len(char) > 2])
    text = re.sub(r'\s+', ' ', text).strip()
    return text

We strip URLs, remove unnecessary symbols, and perform stemming (basically reducing words to their root). It’s like a spa day for text.

Step 4: Choose Your Classifier 🎯

Now for the fun part: picking your weapon of choice! Lets use a couple of simple and a couple of complex algorithm and compare the results. We’ll test out four popular machine learning algorithms:

• Naive Bayes: Quick and simple.
• Support Vector Machine (SVM): Fancy and precise.
• Logistic Regression: Sounds more complex than it is.
• Random Forest: A team of decision trees working together.

Evaluating and Comparing Machine Learning Algorithms

We’ll compare the performance of each algorithm across four key metrics:

1. Execution Time: How long it takes for the algorithm to complete.
2. Accuracy: How many of the predictions are correct overall.
3. Precision: Out of all the predicted positives, how many were actually positive.
4. Recall: Out of all the actual positives, how many were correctly identified.

For each algorithm, we’ll create a function that does the following:

1. Time Calculation: We will track the start and end time of each model’s execution using Python’s time module. This will help us measure the time elapsed, giving us an idea of how efficient the algorithm is in terms of execution speed.
2. TF-IDF Vectorizer: To convert text data into numerical features, we’ll use the TF-IDF (Term Frequency-Inverse Document Frequency) vectorizer. It transforms the raw text into a meaningful vector of features that represent the importance of words in each document relative to the entire dataset.
3. Pipeline Construction: We’ll leverage scikit-learn’s Pipeline to streamline the process. The pipeline will combine the TF-IDF vectorizer and the algorithm (e.g., Naive Bayes, SVM, Logistic Regression, or Random Forest) into a single, cohesive workflow. This makes it easier to manage preprocessing and modeling steps without needing to manually fit and transform the data multiple times.
4. Confusion Matrix and Heatmap: After making predictions with the model, we’ll generate a confusion matrix to visualize how many true positives, true negatives, false positives, and false negatives the model produced. We’ll also create a heatmap using seaborn to visually represent this matrix and identify how well the algorithm distinguishes between good and bad news.
5. Accuracy, Precision, and Recall: These three metrics give deeper insights into the performance of each model:

• Accuracy tells us the overall correctness of the model.
• Precision focuses on how many of the predicted positives are actual positives (useful in situations where false positives are costly).
• Recall tells us how many of the actual positives were correctly identified (critical when missing positives is more concerning).
• By calculating and comparing these metrics, we can evaluate the strengths and weaknesses of each algorithm in different scenarios.

6. Pickle the Model: To save time in the future and avoid retraining the model from scratch, we’ll use the pickle module to store each trained model in a serialized format. This allows us to reuse the model later without needing to repeat the training process.

Let’s look at the code.

a) Multinomial Naive Bayes 🐦

def useMultinominalNB(xTrain, yTrain, xTest, yTest):
    startTime =time.time()
    model = make_pipeline(TfidfVectorizer(max_features=2500), MultinomialNB())
    model.fit(xTrain, yTrain)
    ypred = model.predict(xTest)
    endTime = time.time()
    elapsedTime = endTime-startTime
    cm = confusion_matrix(yTest,ypred)
    accuracy = accuracy_score(ypred, yTest)
    precision = precision_score(yTest,ypred)
    recall = recall_score(yTest,ypred)
    reportSample = {
        'algorithm': 'MultiNominal NaiveBayes',
        'timeTaken': elapsedTime,
        'accuracy': round(accuracy*100,3),
        'precision': round(precision,3),
        'recall': round(recall,3)
    }
    allReport.append(reportSample)

    sns.heatmap(cm,annot=True)
    plt.xlabel('True Values Bad = 0 | Good = 1')
    plt.ylabel('Predicted Values')
    plt.title('Bad News vs Good News NB')
    plt.show()
    pickle.dump(model,open(f"models/{allModel.get('nb','model')}",'wb'))
    return accuracy

With Naive Bayes, we quickly get a result. Spoiler alert: it’s pretty fast, but not always the most accurate.

b) Support Vector Machine: Grid Search Edition ⚙️

SVM’s like the fancy gadget of the bunch. It uses a thing called Grid Search to find the best parameters for us. Think of it as an algorithm on steroids.

def useSVCGridSearch(xTrain,yTrain,xTest,yTest):
    startTime = time.time()
    pipe = make_pipeline(TfidfVectorizer(max_features=2500), SVC())
    param_grid = {
        'svc__kernel': ['linear', 'rbf', 'poly', 'sigmoid'],
        'svc__degree': [2, 3, 4, 5],
        'svc__gamma': [0.001, 0.01, 0.1, 1, 10, 100],
        'svc__C': [0.0001, 0.001, 0.01, 0.1, 1, 10, 100]
    }

    grid = GridSearchCV(pipe,param_grid,cv=5)
    grid.fit(xTrain, yTrain)
    model = grid.best_estimator_
    print(grid.best_score_)
    model.fit(xTrain,yTrain)
    ypred = model.predict(xTest)
    endTime = time.time()
    elapsed = endTime-startTime
    accuracy = accuracy_score(ypred, yTest)
    precision = precision_score(yTest, ypred)
    recall = recall_score(yTest, ypred)
    reportSample = {
        'algorithm': 'SVM With Grid Search',
        'timeTaken': elapsed,
        'accuracy': round(accuracy * 100, 3),
        'precision': round(precision, 3),
        'recall': round(recall, 3)
    }
    allReport.append(reportSample)

    cm = confusion_matrix(yTest, ypred)
    sns.heatmap(cm, annot=True)
    plt.xlabel('True Values Bad = 0 | Good = 1')
    plt.ylabel('Predicted Values')
    plt.title('Bad News vs Good News SVC')
    plt.show()
    pickle.dump(model,open(f"models/{allModel.get('svc','model2')}",'wb'))
    return accuracy

Grid Search

In machine learning, tuning hyperparameters can significantly affect a model’s performance. Grid Search is a method for finding the optimal hyperparameters by trying different combinations and cross-validating their performance. It’s a bit like fine-tuning an engine until it performs at its best.

With SVM, we sacrifice a bit of time for accuracy. Sometimes, good things take time, right?

c) Random Forest 🌲🌲

It’s not just a tree — it’s a forest of decision trees! This algorithm tends to be accurate but might take a little longer to train.

def useRandomGridSearch(xTrain,yTrain,xTest,yTest):
    startTime=time.time()
    pipe = make_pipeline(TfidfVectorizer(max_features=2500), RandomForestClassifier())
    param_grid = {
        'randomforestclassifier__n_estimators': [100, 200, 300],  # Number of trees in the forest
        'randomforestclassifier__max_depth': [None, 10, 20, 30],  # Depth of each tree
        'randomforestclassifier__min_samples_split': [2, 5, 10],  # Minimum number of samples to split a node
        'randomforestclassifier__min_samples_leaf': [1, 2, 4],  # Minimum number of samples required to be a leaf node
        'randomforestclassifier__bootstrap': [True, False]  # Whether bootstrap samples are used when building trees
    }

    grid = GridSearchCV(pipe,param_grid,cv=5)
    grid.fit(xTrain, yTrain)
    model = grid.best_estimator_
    print(grid.best_score_)
    model.fit(xTrain,yTrain)
    ypred = model.predict(xTest)
    endTime = time.time()
    elapsed = endTime - startTime
    accuracy = accuracy_score(ypred, yTest)
    precision = precision_score(yTest, ypred)
    recall = recall_score(yTest, ypred)
    reportSample = {
        'algorithm': 'RandomForest With Grid',
        'timeTaken': elapsed,
        'accuracy': round(accuracy * 100, 3),
        'precision': round(precision, 3),
        'recall': round(recall, 3)
    }
    allReport.append(reportSample)
    cm = confusion_matrix(yTest, ypred)
    sns.heatmap(cm, annot=True)
    plt.xlabel('True Values Bad = 0 | Good = 1')
    plt.ylabel('Predicted Values')
    plt.title('Bad News vs Good News Random Forest')
    plt.show()
    pickle.dump(model,open(f"models/{allModel.get('rf','model2')}",'wb'))
    return accuracy

If you want a robust solution, this one’s your go-to. Forests are resilient, after all!

d) Logistic Regression 🚀

Last but not least, good ol’ Logistic Regression. Don’t be fooled by its name — it’s simple and gets the job done!

def useLogisticRegression(xTrain,yTrain,xTest,yTest):
    startTime =time.time()
    model = make_pipeline(TfidfVectorizer(max_features=2500), LogisticRegression(max_iter=1000))
    model.fit(xTrain, yTrain)
    ypred = model.predict(xTest)
    endTime = time.time()
    elapsed = endTime - startTime
    accuracy = accuracy_score(ypred, yTest)
    precision = precision_score(yTest, ypred)
    recall = recall_score(yTest, ypred)
    reportSample = {
        'algorithm': 'SVM With Logistic',
        'timeTaken': elapsed,
        'accuracy': round(accuracy * 100, 3),
        'precision': round(precision, 3),
        'recall': round(recall, 3)
    }
    allReport.append(reportSample)
    accuracy = accuracy_score(ypred, yTest)
    cm = confusion_matrix(yTest, ypred)
    sns.heatmap(cm, annot=True)
    plt.xlabel('True Values Bad = 0 | Good = 1')
    plt.ylabel('Predicted Values')
    plt.title('Bad News vs Good News Logistic Regression')
    plt.show()
    pickle.dump(model,open(f"models/{allModel.get('lr','model2')}",'wb'))
    return accuracy

It’s like the dependable friend who always delivers. No bells and whistles, just results.

A few more lines of code to link up everything.

largeDataSet = False
if largeDataSet==True:
    newsData = pd.read_csv('TrainingDataNewsLargeDataSet.csv')
else:
    newsData = pd.read_csv('TrainingDataNews.csv')

getDataAnalysis(newsData)

# Split the data into target and Feature
X = newsData['news']
y = newsData['label']

# Initializing the stemmer
stemmer = LancasterStemmer()
# getting the stopwords
stopwords = set(stopwords.words('english'))

# using apply since it is series
X = X.apply(processCleanData)

# Splitting the data into train and testing
xTrain, xTest, yTrain, yTest = train_test_split(X, y, train_size=0.7, random_state=123)

print(
    f'The accuracy with Multinominal NB {round(useMultinominalNB(xTrain, yTrain, xTest, yTest) * 100)}%')
print(
    f'The accuracy with Logistic Regression  {round(useLogisticRegression(xTrain, yTrain, xTest, yTest) * 100)}%')

print(
    f'The accuracy with SVC  {round(useSVCGridSearch(xTrain, yTrain, xTest, yTest) * 100)}%')
print(
    f'The accuracy with Random Forest  {round(useRandomGridSearch(xTrain, yTrain, xTest, yTest) * 100)}%')

#Creating a final report for the algorithms above
finalReportData = pd.DataFrame(allReport)
finalReportData.columns = ['Algorithm Name','Time Taken (seconds)','Accuracy','Precision','Recall']
print(finalReportData)

Wow, that was quite a bit of work! Now, let’s check out the results. I was expecting the more complex algorithms, like SVC and Random Forest, to perform the best.

For less Data

Hold on a minute — Random Forest took quite a while to run (This is because it has a lot of hyperparameters to work on) . But wait, what’s this? The accuracy of the more complex algorithms (Random Forest and SVC) is actually much lower compared to the simpler ones (Naive Bayes and Logistic Regression). What’s going on here

Why Simpler Algorithms Shine on Smaller Datasets ✨

You might wonder why Logistic Regression and Naive Bayes seem to outperform the fancier models on smaller datasets. Here’s why:

• Bias-Variance Tradeoff: Both Logistic Regression (LR) and Naive Bayes (NB) are high-bias, low-variance models. This means they’re less prone to overfitting, which is a common issue with small datasets. LR assumes a linear relationship between features and the target, while NB calculates conditional probabilities, simplifying the model.
• Sample Efficiency: LR and NB require fewer data points to converge. They can extract meaningful patterns from smaller datasets, while models like Random Forest (RF) and SVM tend to need more data to capture complex decision boundaries.
• Feature Space Simplicity: In text classification tasks (like ours, using TF-IDF), the number of features often exceeds the number of samples. Since Naive Bayes calculates probabilities independently for each feature, it handles this imbalance well. Logistic Regression, being a linear classifier, also works efficiently in these scenarios.
• Complex Models Overfitting: RF and SVM are powerful but can overfit when the dataset is small. They might learn patterns from noise, leading to poor generalization. RF splits the data across multiple trees, which becomes unreliable when features are irrelevant, while SVM tends to overfit easily without enough data to capture true patterns.

In short, simpler algorithms like Logistic Regression and Naive Bayes perform well on small datasets because they are less prone to overfitting and make the most of limited data. More complex models, like Random Forest and SVM, need larger datasets to avoid overfitting and to fully showcase their strengths.

Alright, so what if we increase the dataset size? To test this, just set the largeDataSet variable to True and remember to comment out the code that removes duplicates in the data analysis function. This process will take some time, so hang tight and be patient.

For large data set

That took a while. Now, we can see that the more complex algorithms are performing significantly better as well as the simpler ones.

Step 5: Predictions

Now that we’ve trained our models, let’s have some fun with predictions.

def predictData(dataToPredict,model):
    dataToPredictCleaned = list(map(processCleanData,dataToPredict))
    model = pickle.load(open(f'models/{model}', "rb"))
    predicted = model.predict(dataToPredictCleaned)
    print(predicted)

dataToPredict = ["10 People killed in card accident","5 people awarded with 10 million"]

predictData(dataToPredict,allModel.get('lr'))

We’ve set up a straightforward function that takes an array of news headlines and the model we’ve trained. We stored the trained model in a pickle file, so it’s easy to use now.

Just make sure to pass the headlines through our processCleanData function to clean them before making predictions. The output of this function will be [0, 1], where 0 stands for "bad" and 1 stands for "good"—and it’s spot on!

And there you have it! With a solid understanding of both the code and the theory behind these models, you’re all set to predict the next big news headline!

As always the full code is available in my Github!

Follow For More!!