BBC NEWS CATEGORIZATION WITH NLP

Natural Language Processing (NLP) is a branch of artificial intelligence that focuses on enabling computers to understand, interpret, and generate human language. It involves various techniques to process and analyse large amounts of natural language data, such as text or speech, to extract meaningful information. NLP is widely used for tasks like language translation, sentiment analysis, text summarization, and speech recognition.

The use of NLP (Natural Language Processing) is essential because it enables computers to understand, interpret, and generate human language, bridging the gap between human communication and machine understanding. This technology has numerous applications across different fields, making it a key component in modern AI-driven solutions. Some of the main areas where NLP is used include:

· Text Classification: Categorizing emails, documents, and news articles automatically.

· Sentiment Analysis: Analyzing customer reviews or social media posts to gauge public opinion.

· Machine Translation: Translating text from one language to another.

· Speech Recognition: Converting spoken language into text, used in virtual assistants like Siri and Alexa.

· Chatbots and Virtual Assistants: Providing real-time, automated customer support or information retrieval.

· Text Summarization: Condensing long articles into shorter summaries for quick information consumption.

· Named Entity Recognition (NER): Identifying and classifying entities like names, places, and dates in a text.

· Information Retrieval: Enhancing search engines to provide more relevant results based on user queries.

1. Introduction

This project aims to classify news articles from the BBC News dataset using advanced Natural Language Processing (NLP) techniques. The primary goal is to explore how different models can be utilized to efficiently group news articles into relevant categories. The process began with exploratory data analysis (EDA) and comprehensive preprocessing, followed by the implementation of multiple modelling approaches, each tested with various parameters to optimize performance. The results were analysed using a combination of coherence scores, reconstruction errors, and visualization techniques to determine the best-performing models.

2. Dataset Overview

The dataset comprised 35,085 news articles, following the removal of duplicates from an initial set of 39,483 entries. Initial steps included cleaning duplicate records and handling missing values. Furthermore, the publication dates of the articles were converted into a datetime format to examine the distribution of news over the years and months, which provided insights into major events and trends across different time periods.

Dataset information

3. Preprocessing

The preprocessing pipeline involved below key steps:

· Stop-words and Non-Alphabetic Characters Removal: To focus on the most meaningful words, common stop-words and non-alphabetical symbols were removed.

· POS Tagging and Lemmatization: Part-of-Speech tagging helped in understanding the grammatical structure, while lemmatization ensured that different forms of a word were reduced to a single canonical form.

· Entity Recognition and Additional Stop-words Filtering: Named Entity Recognition (NER) was used to identify and exclude specific named entities like persons, organizations, and locations to avoid skewing the topic categorization. Additionally, suffixes and extra stop-words were eliminated to refine the dataset further.

4. Modelling Techniques

The core of the project was to experiment with different models to categorize the news effectively. Three main approaches were explored:

4.1. Non-Negative Matrix Factorization (NMF)

NMF was employed to decompose the dataset into a set of topics and assign each document to these topics. The technique relies on matrix factorization, making it easier to interpret the word-topic and document-topic matrices. Various parameter settings were tested, adjusting the number of topics, maximum features, and top words per category.

4.1.1. Experiment 1

The visual presents the results of a categorization study using different maximum feature configurations. Each configuration shows the number of categories and their corresponding coherence scores. Five configurations were compared: 15000, 10000, 5000, 3500, and 2500 maximum features.

1. Best Coherence Score: The highest coherence score (0.4991) was achieved with the “Max features: 5000” configuration and 9 categories. This suggests that this configuration produced more consistent and meaningful categories compared to others.
2. Coherence Range: The coherence scores across the five configurations generally range from 0.434 to 0.499. This indicates a consistent level of coherence, but it also highlights the need to identify an optimal configuration.

The analysis compared different configurations to find the optimal categorization results, and the 5000 max features configuration with 9 categories achieved the best coherence score. Overall, it seems that configurations with a large number of features did not provide a significant advantage, suggesting that similar results can be obtained with fewer features.

4.1.2. Experiment 2

Consistency:

Min: 0.05 — Max: 0.85 and Min: 0.05 — Max: 0.80 are consistent, providing the same high coherence score across various categories, with 10 categories standing out.

Variability:

Min: 0.01 — Max: 0.85 and Min: 0.01 — Max: 0.80 settings show more variability, with coherence scores decreasing notably as the number of categories changes, indicating less stability.

Relatively high and consistent coherence score 0.4489 for 10 categories emerge as the optimal choice, balancing coherence and interpretability effectively.

4.1.3. Experiment 3

With experiment 3, the results are obviously better compared to the previous 2 experiments. If we will go with NMF, it is obvious that we will select above parameters to proceed.

For further understanding, we want to see the PCA (Principal Component Analysis) to reduce the high-dimensional TF-IDF matrix to just 2 components for easier visualization. By projecting the data into 2 dimensions, we can visually inspect how well the NMF model has clustered the data. We select top 5 categories in the above coherence scores:

· 3 categories: Displays clear, distinct clusters, suggesting a natural division into three broad themes. However, this might oversimplify the data, losing finer details within the articles.

Below you can see how the data set divided into 3 new categories:

In one category having 70% news is clearly not we want !

· 9–8–10 categories show more overlaps and the boundaries of the topics are less clear.

· Let’s have a look to the 7 category version:

7-category configuration appears to provide sufficient detail while maintaining relatively clear separations, making it an effective choice for categorizing news articles. It avoids the oversimplification seen with 3 categories and the over-segmentation observed with 8,9 or 10 categories, thus presenting a middle-ground solution that captures nuanced themes.

We select 7 categories for NMF and we want to see the identifying words in each category:

We decide to name the categories as below:

4.2. Latent Dirichlet Allocation (LDA)

LDA is a generative statistical model used for topic modelling in natural language processing used to find hidden topics in a bunch of text documents. It assumes each document is about several topics, and each topic is made up of specific words.

How LDA Works:

Documents as Mixtures: Each product review can be a mix of topics like “product quality,” “customer service,” and “price.”

Topics as Word Distributions: The “product quality” topic might have words like “durable,” “reliable,” “defective,” and “poorly made.” The “customer service” topic might have words like “helpful,” “responsive,” “rude,” and “unhelpful.”

Probabilistic Approach: LDA would analyse the words in each review and assign probabilities to different topics. For example, a review that mentions a product’s great features and excellent customer support would likely be classified as a mixture of “product quality” and “customer service.”

In our study, 2 experiments with LDA focused on optimizing parameters like Alpha (α) and Beta (β), which control the sparsity of the document-topic and topic-word distributions, respectively.

Modelling on 7 categories with LDA with default parameters:

Perplexity: -9.0035, Coherence Score: 0.3413

1. Separation of Topics: Intertopic Distance Map

• Moderate level of separation (distinct but not entirely independent).

2. Saliency of Terms (Top-30 Most Salient Terms)

• The model is capturing important and frequent terms.

• Terms like “win” appearing across multiple topics indicates reduced sharpness of differentiation.

3. Coherence Score (34%)

Model’s topics are somewhat interpretable, but this score is relatively low.

Modelling on 7 categories with LDA with adjusted Alpa, ETA and Passes Parameters:

Alpha and Eta Parameters:

• In the first code block, the LdaModel does not specify alpha and eta parameters, which means the default settings are used.
• In the second code block, alpha and eta are set to “auto”, allowing the model to automatically learn optimal values, potentially leading to better fitting.

Passes Parameter:

• In the first code block, the passes parameter is not used, which likely defaults to 1, meaning the model runs only one pass over the data.
• In the second code block, passes=10 is explicitly set, enabling the model to perform multiple passes, which helps improve learning.

1. Intertopic Distance Map:

• Larger topics (1, 2, 3, and 4) are more distinctly separated.

• Smaller topics (5, 6, and 7) remain close and overlapped.

2. Saliency of Terms (Top-30 Most Salient Terms)

• The model is capturing important and frequent terms.

• However, terms like “win” appearing across multiple topics indicates reduced sharpness of differentiation.

3. Coherence improvement from 0.34 to 0.42 indicates that the topics generated by the LDA model are more consistent and meaningful. Coherence measures how interpretable and semantically related the words within a topic are.

The perplexity score being negative might seem unusual, but it typically results from how perplexity is calculated in topic modelling. Perplexity in topic models often involves the log likelihood of the data under the model, which can be negative when the probability values are less than 1 (common in probabilistic models). Since log probabilities are negative, perplexity scores in some implementations are also expressed as negative values, with a higher negative number (e.g., -9.0035) indicating lower perplexity and better performance.

• Old Results (Perplexity = -9.0035) had a lower perplexity (better model performance).
• Tuned Results (Perplexity = -8.7575) had a slightly higher perplexity, meaning the model may have lost some predictive accuracy on new data.

While LDA model showed moderate success in separating topics, the coherence scores were generally lower compared to NMF, indicating that the model’s topics were less interpretable. Further tuning improved the scores slightly, but overall, LDA was outperformed by NMF in terms of topic clarity.

4.3. BERT Embeddings with K-Means Clustering

BERT Embeddings:

BERT is a sophisticated language model created by Google. It grasps the context of words in a sentence by examining them bidirectionally, meaning it considers both left-to-right and right-to-left context.

It transforms text (be it words, phrases, or full sentences) into high-dimensional numerical vectors called embeddings. These embeddings encapsulate the semantic meaning of the text, so that similar words or sentences will have similar embeddings.

K-Means Clustering:

K-Means is a well-known unsupervised machine learning algorithm that clusters data into K-Means Clusters. It groups similar data points (such as embeddings) based on their proximity to a central point.

When applied to BERT embeddings, K-Means can cluster texts with similar meanings together, even if they do not share exact words. This makes it a great tool for tasks like topic clustering, document classification, and identifying semantic similarities.

In our study, we employed BERT embeddings alongside K-Means clustering to group similar news articles. We evaluated the embeddings using methods like the elbow method and PCA for dimensionality reduction.

Silhouette Score: 0.04051

Silhouette Score measures the quality of clustering, particularly how similar an object is to its own cluster (cohesion) compared to other clusters (separation). The score ranges from -1 to 1:

• 1: Perfectly clustered, well-separated clusters.
• 0: Overlapping clusters, no clear separation.
• -1: Wrongly clustered, objects closer to other clusters than their own.

In this case, a silhouette score of 0.04 suggests that the clusters are not very distinct and there is significant overlap.

The elbow method is somewhat inconclusive in this case because there is no sharp “elbow” visible. While 3 to 5 clusters could still be tested based on the slight bend, the lack of a clear elbow suggests that choosing the number of clusters may require further experimentation or consideration of other factors, such as domain knowledge or silhouette scores to assess the quality of clustering.

The degree of separation might suggest that while the clustering captures some structure, there might be opportunities to improve clarity, such as by tweaking the number of clusters, adjusting BERT embeddings, or refining the feature extraction process.

Despite the power of BERT in capturing contextual nuances, this method did not perform as well as anticipated due to the lack of clear boundaries between clusters.

5. Evaluation and Analysis

The models were primarily evaluated based on coherence scores and visual inspections using PCA. NMF consistently demonstrated better performance, achieving higher coherence scores and more distinct separation between topics. Specifically, configurations involving 7 categories, 8 top words, and 5000 maximum features delivered the most balanced results, ensuring both interpretability and performance:

LDA and BERT clustering, on the other hand, showed limitations in topic distinction and required more tuning to improve their effectiveness.

6. Conclusion

The project concluded that NMF was the most effective technique for categorizing BBC News , balancing performance, interpretability, and clarity. It generated higher coherence scores, providing clearer and more distinct topic clusters that could be easily labelled.

Compared to probabilistic and embedding-based approaches, NMF’s straightforward and interpretable structure allowed for efficient and reliable news categorization.

This comprehensive analysis was made possible through the collaborative efforts of Mehmet Emre Öz, Melike Sevinç and myself. Their insights and contributions were invaluable. Stay tuned for more insights and analyses. Until the next challenge!

Gizem Özmen