Sentiment STAR-ity: Evaluating Review-Score Consistency for Smarter Shopping

In the era of e-commerce, customer reviews play a pivotal role in aiding potential buyers in their decision-making process. Star ratings have become a widely accepted metric for summarizing customer sentiment towards a product. However, the relationship between sentiment expressed in text reviews and the corresponding star ratings remains an essential aspect to explore.

This research aims to investigate the relationship between the sentiment expressed in customer reviews and the corresponding star ratings with the overarching goal of evaluating the usefulness and efficacy of existing reviews to potential buyers.

Hypothesis

We hypothesize that the sentiment of the reviews will align with the star rating given by the customer. Higher star ratings will correlate with predominantly positive sentiment, while lower star ratings will align with predominantly negative sentiment.

Data

The dataset has been obtained from the UCSD Amazon Reviews repository. The dataset provides information on features such as the overall star rating, user verification, review time, reviewer name, asin, summary, unix review time, vote, style, image.

It includes 371,345 reviews.

Here’s a look at the data frame:

Some important features we focused on:

1. overall — Star rating assigned by the user between 1.0–5.0 (Numeric)
2. verified — User verification status (True or False)
3. reviewerID
4. reviewText — Review assigned by the user to the product

Methodology

A. Data Selection, Analysis and Preprocessing

Started data cleaning by sorting through the number of reviews given by the user compared to the review score on a 5pt star scale.

At this point, sorting the data by user verification was crucial.

Without having eliminated fake reviews, the vast majority of reviews were positive with a ranking of 5, followed by a ranking of 4.

We assigned the polarity values based on the following criteria:

• if the overall star rating was between 0–1, it was assigned a negative value (0)
• if the overall star rating was between 2–3, it was assigned a neutral value (1)
• if the overall star rating was between 4–5, it was assigned a positive value (2)

B. Tokenization

By leveraging spaCy’s language-specific tokenization algorithms, we were able to efficiently break down sentences into individual tokens, allowing for easier analysis and feature extraction.

C. Vocabulary building and Embedding

We focused on utilizing Scikit-learn to build a robust vocabulary and implement word embedding techniques. Scikit-learn’s functionality allowed us to efficiently create a vocabulary by extracting unique tokens from the text corpus, enabling numerical representation of words for our machine learning algorithms.

Additionally, Scikit-learn’s word embedding capabilities helped us obtain dense vector representations of words, capture semantic relationships and enhance the model’s understanding of word context, which ultimately improved its performance in text classification and sentiment analysis.

D. Model Development

We opted for a straightforward approach using simple logistic regression for both training and prediction data. By employing Scikit-learn’s logistic regression implementation, we were able to efficiently train the model on our labeled dataset and predict outcomes for new, unseen text samples.

Looking good so far!

Let’s take a look at the results.

Results

In line with our hypothesis, the model was able to predict the sentiment given unseen test reviews.

Precision evaluates the accuracy of positive predictions, while recall assesses the model’s ability to capture all positive instances in the dataset.

The F1 score is a crucial machine learning evaluation metric that takes into account both precision and recall.

In contrast, the accuracy metric measures the overall correctness of predictions made by the model across the entire dataset.

Our model achieved an impressive 84% accuracy and a macro average f1 score of 68%.

Conclusion

In conclusion, the model was able to predict the sentiment given unseen test reviews with an 84% accuracy. By quantifying the percentage of reviews where the sentiment matched the assigned star rating, we were able to assess the usefulness and efficacy of existing reviews for other potential buyers.

For future work, we will investigate the hypothesis that certain words in reviews will contribute significantly to the overall sentiment prediction.

We will also try to build a more sophisticated sentiment analysis model by exploring transfer learning.

References

1. Amazon dataset:

a. https://cseweb.ucsd.edu/~jmcauley/datasets/amazon_v2/

2. Twitter Sentiment Analysis Template:

a. https://colab.research.google.com/github/NeuromatchAcademy/course-content- dl/blob/main/projects/NaturalLanguageProcessing/sentiment_analysis.ipynb

3. Literary references:

a. https://paperswithcode.com/paper/sentiment-analysis-based-on-deep-learning-a

b. https://spacy.io/usage/linguistic-features

Closing note

A special thanks to my peers Jimmy ibañez, Arshia Gharooni, and Hector Julian Gutierrez Hoyos who put endless hours into getting this project to the stage it is currently at.

This project would be incomplete without our group TA, Tunde Ajayi’s exceptional patience, dedicated teaching, and going above and beyond by staying extra hours to make this project possible. His unwavering support has been instrumental in our success, and we truly appreciate all the effort he put in to help us achieve our goals. Thank you!

A sincere appreciation to our project TA Fu-Te Wong and mentor Anna Ivanova and all other Neuromatch Academy organizers involved for their invaluable guidance and support throughout our project.