Phishing email detection using NLP — A guided project

Executive Summary

In today’s digital era, email phishing remains one of the most pervasive threats. Cybercriminals use deceptive emails to compromise sensitive information, causing significant damage. This proposal outlines a project to develop a lightweight phishing email detection system leveraging Natural Language Processing (NLP). The system will analyze email content and metadata to identify malicious patterns and classify emails as legitimate or phishing.

By implementing this project with minimal resources, we aim to demonstrate the potential of AI-driven cybersecurity tools using open-source technologies and freely available datasets.

Introduction

Phishing attacks exploit human vulnerabilities by impersonating trusted entities to extract confidential information. Using advancements in AI and NLP, even individuals with limited resources can counter such attacks effectively. This project focuses on developing a cost-effective solution to detect and mitigate phishing emails.

Objectives

1. Build a system capable of classifying emails as phishing or legitimate with high accuracy.
2. Analyze email content, subject lines, and metadata for suspicious patterns.
3. Develop the solution using free and open-source tools.

Methodology

1. Data Collection: Use freely available datasets such as the ENRON and Kaggle phishing datasets. Link for the dataset used for this project: https://www.kaggle.com/datasets/subhajournal/phishingemails?resource=download
2. Data Preprocessing:

• Remove irrelevant information (e.g., HTML tags, URLs, and special characters).
• Tokenize, lemmatize, and vectorize email content using open-source libraries like NLTK and Scikit-learn.

1. Feature Engineering: Extract features like word count, presence of keywords (e.g., “urgent,” “click here”), and metadata analysis.
2. Model Development:

• Use simple machine learning models such as Logistic Regression or Naïve Bayes.
• Evaluate models based on precision, recall, and F1-score.

1. Testing: Evaluate the model on a portion of the dataset to ensure performance.

Expected Outcomes

1. A lightweight phishing detection system capable of identifying phishing emails with reasonable accuracy.
2. Enhanced understanding of how NLP techniques can be applied to cybersecurity.
3. Practical demonstration of building AI-driven solutions on a limited budget.

Budget

1. Tools:

• Python and Jupyter Notebook (free).
• Open-source libraries: pandas, nltk, scikit-learn, and matplotlib.

1. Datasets: Freely available online.
2. Hardware: A personal computer or laptop with basic processing power.

Estimated total: $0 (leveraging existing resources and open-source tools).

Explanation for Non-Technical Audience

Emails are one of the most common ways that hackers try to trick people. Sometimes, these emails pretend to be from a trusted company or a friend but are actually designed to steal important information, like passwords or credit card numbers. This is called phishing.

Our project aims to build a smart system that can recognize these fake emails. Using technology called Natural Language Processing (NLP), the system will “read” the email’s content and look for signs of trickery, such as strange language or suspicious links. It will then decide if the email is real or fake.

Once we create this system, it can help protect everyone by automatically stopping phishing emails before they cause harm. This means better safety for our information and less risk of falling victim to online scams.

Why Logistic Regression is Used

Logistic Regression is commonly used for classification tasks, particularly binary classification, where the goal is to categorize data into one of two classes. Here’s why it is an excellent choice for many machine learning tasks, including phishing email detection:

1. Simplicity and Interpretability:

• Logistic regression is a simple and interpretable algorithm. It calculates the probability of a data point belonging to a specific class, making it easy to understand and explain.

• For instance, it can tell you not only whether an email is phishing or not but also how confident the model is in its prediction.

2. Probability-Based Classification:

• Logistic regression predicts the probability of a given data point belonging to a class using the sigmoid function. If the probability exceeds a threshold (typically 0.5), the model classifies the instance into one class; otherwise, it goes to the other.

• This probabilistic approach is useful in phishing detection because it allows you to quantify the certainty of a prediction.

3. Effective for Linearly Separable Data:

• Logistic regression works well when there’s a linear relationship between the input features and the log-odds of the outcome. While phishing detection is often complex, it can still benefit from logistic regression in the presence of linearly separable patterns in the dataset.

4. Low Computational Complexity:

• Logistic regression is computationally efficient compared to more complex models like neural networks. This makes it suitable for projects where quick model training and deployment are important.

5. Baseline Model:

• Logistic regression is often used as a baseline model in machine learning projects because it provides a strong foundation for comparison. If more complex models don’t significantly outperform logistic regression, it’s usually preferred for its simplicity and efficiency.

What is Logistic Regression?

Logistic Regression is a statistical method used to model the relationship between a dependent variable (target) and one or more independent variables (features). Despite its name, it’s a classification algorithm, not a regression one.

Key Concepts:

1. Sigmoid Function:

• The core of logistic regression is the sigmoid function, which maps any real-valued number to a range between 0 and 1:

[embed]

Here, z = w . x + b, where w represents weights, x is the input, and b is the bias.

• The output of the sigmoid function represents the probability that a data point belongs to a particular class.

2. Decision Boundary:

• Logistic regression establishes a decision boundary to separate classes. For binary classification, it uses a threshold (commonly 0.5) to assign data points to one of the two classes.

3. Log-Loss Function:

• Logistic regression uses the log-loss function to optimize its parameters during training. This function penalizes incorrect predictions more heavily when the model is very confident in the wrong outcome:

[embed]

• y,sub,i : Actual class (0 or 1)
• p,sub,i: Predicted probability of the class

4. Linear Relationship in the Log-Odds:

• Logistic regression assumes that the log-odds (logarithm of the odds ratio) of the dependent variable can be expressed as a linear combination of the features:

[embed]

This linear relationship allows the model to effectively separate classes when the data is linearly separable.

Use of Logistic Regression in Phishing Email Detection:

In phishing detection:

• Input Features: The input features could include characteristics like email length, presence of suspicious links, use of certain keywords, or sender address reputation.

• Output: Logistic regression predicts whether an email is “phishing” (class 1) or “safe” (class 0).

• Benefits:

• It provides a quick and interpretable solution for detecting patterns in phishing emails.

• The model’s probabilities can help set thresholds for classification (e.g., a stricter threshold might classify emails as phishing only if the probability exceeds 0.8).

Logistic regression’s efficiency and ability to provide probabilities make it an excellent starting point for phishing detection tasks. If combined with techniques like feature engineering and text preprocessing (e.g., TF-IDF), it can yield strong results.

To learn more about logistic regression, follow the links:

https://aws.amazon.com/what-is/logistic-regression/

https://www.geeksforgeeks.org/understanding-logistic-regression/

Step-by-Step Instructions:

You can also check out my GitHub repo at:

https://github.com/CodenameUnknownHobbes/Phishing-Email-Detection-using-NLP.git

Step 1: Project Setup

1. Install Required Tools

• Ensure you have Python 3.x installed on your system.
• Install Jupyter Notebook:

pip install notebook

• Install essential Python libraries:

pip install pandas numpy scikit-learn nltk matplotlib

2. Set Up the Environment

Create a project directory:

mkdir phishing_email_detection
cd phishing_email_detection

Start Jupyter Notebook:

jupyter notebook

3. Download Datasets

• Obtain datasets from sources like Kaggle (e.g., “Phishing Email Dataset”) or the ENRON dataset.
• Place the dataset in the project directory.

Step 2: Import all modules

import pandas as pd
import numpy as np
import re
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
from nltk.stem import WordNetLemmatizer
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report, confusion_matrix, accuracy_score
import matplotlib.pyplot as plt
import seaborn as sns

Module Imports: This code imports necessary libraries for data manipulation (pandas, numpy), text processing (re, nltk), machine learning (sklearn), and visualization (matplotlib, seaborn).

Step 3: Load the datasets

# Load the dataset

df = pd.read_csv('Phishing_Email.csv')

Loading Data: The dataset is loaded into a pandas DataFrame using pd.read_csv, allowing for easy manipulation and analysis.

Step 4: Refinement of the datasets

# Drop unnecessary columns and rename for clarity

df = df.drop(columns=['Unnamed: 0'], errors='ignore')
df = df.rename(columns={'Email Text': 'email_text', 'Email Type': 'label'})
# Handle missing values

df['email_text'] = df['email_text'].astype(str)
df['email_text'] = df['email_text'].fillna('')

df['label'] = df['label'].map({'Safe Email': 0, 'Phishing Email': 1})

Data Cleaning: Unnecessary columns are dropped, and columns are renamed for clarity. This prepares the data for further processing.

Handling Missing Values: Missing values in the email text are filled with empty strings, and labels are converted to binary format (0 for safe emails, 1 for phishing).

Step 5: Data exploration

# Data exploration

print("Dataset shape:", df.shape)
print("Label distribution:")
print(df['label'].value_counts())

Output:

Dataset shape: (18650, 2)
Label distribution:
0    11322
1     7328
Name: label, dtype: int64

Exploring Data: This code prints the shape of the dataset (number of rows and columns) and counts the distribution of labels, providing insight into class balance.

Step 6: Visualization of label distribution

sns.countplot(x='label', data=df)
plt.title('Label Distribution')
plt.xlabel('Label (0: Safe, 1: Phishing)')
plt.ylabel('Count')
plt.show()

Output

Visualizing Distribution: A count plot visualizes how many emails fall into each category (safe vs. phishing), helping to identify any imbalances.

Step 7: Preprocessing function

# Preprocessing function

stop_words = set(stopwords.words('english'))
lemmatizer = WordNetLemmatizer()

def clean_text(text):
    text = re.sub(r'<[^>]+>', '', text)  # Remove HTML tags
    text = re.sub(r'https?://\S+', '', text)  # Remove URLs
    text = re.sub(r'\W', ' ', text)  # Remove special characters
    tokens = word_tokenize(text.lower())
    tokens = [lemmatizer.lemmatize(word) for word in tokens if word not in stop_words]
    return ' '.join(tokens)

Text Cleaning Function: This function removes HTML tags, URLs, and special characters from emails. It tokenizes the text, converts it to lowercase, removes stop words, and lemmatizes words to their base forms.

Example:

Original: “Click here to win $1000!!!”

Cleaned: “Click here to win”

Stop Words Removal: Eliminate common words (e.g., ‘and’, ‘is’, ‘to’) that do not contribute significant meaning.

Example:

[‘Click’, ‘here’, ‘win’]

Lemmatization: Reduce words to their base or root form to ensure uniformity.

Example:

‘winning’, ‘wins’ → ‘win’

Step 8: Apply preprocessing

# Apply preprocessing

df['cleaned_text'] = df['email_text'].apply(clean_text)

Applying Preprocessing: The cleaning function is applied to each email in the dataset, creating a new column with cleaned text ready for analysis.

Step 9: Train, Test and split

# Train-test split

X = df['cleaned_text']
y = df['label']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

Splitting Data: The dataset is divided into training and testing sets (80% training, 20% testing) to evaluate model performance effectively.

Step 10: TF-IDF vectorization

# TF-IDF Vectorization

tfidf = TfidfVectorizer(max_features=5000)
X_train_tfidf = tfidf.fit_transform(X_train)
X_test_tfidf = tfidf.transform(X_test)

Feature Extraction: TF-IDF vectorization transforms the cleaned text into numerical features that represent how important a word is in relation to a document while considering its frequency across all documents.

Convert the processed text into numerical representations suitable for machine learning algorithms:

TF-IDF Vectorization: Transform text data into numerical features by evaluating the importance of words within the corpus.

Explanation:

Term Frequency (TF): Measures how frequently a word appears in a document.

Inverse Document Frequency (IDF): Assesses how unique or rare a word is across all documents.

TF-IDF Score: Calculated as TF multiplied by IDF, indicating the significance of a word in a document relative to the entire corpus.

Step 11: Model training

# Model training

model = RandomForestClassifier(n_estimators=100, random_state=42)
model.fit(X_train_tfidf, y_train)

Output:

RandomForestClassifier(random_state=42)

Training Model: A Random Forest classifier is created and trained on the TF-IDF features from the training set to learn how to classify emails as phishing or safe.

Step 11: Model evaluation

y_pred = model.predict(X_test_tfidf)
print("Accuracy:", accuracy_score(y_test, y_pred))
print("Confusion Matrix:")
print(confusion_matrix(y_test, y_pred))
print("Classification Report:")
print(classification_report(y_test, y_pred))

Output:

Accuracy: 0.9621983914209116
Confusion Matrix:
[[2194   79]
 [  62 1395]]
Classification Report:
              precision    recall  f1-score   support

           0       0.97      0.97      0.97      2273
           1       0.95      0.96      0.95      1457

    accuracy                           0.96      3730
   macro avg       0.96      0.96      0.96      3730
weighted avg       0.96      0.96      0.96      3730

Evaluating Performance: The model’s predictions on the test set are compared against actual labels to calculate accuracy. A confusion matrix and classification report provide detailed insights into model performance across different metrics.

• Performance Metrics:

• Accuracy: Proportion of correctly classified emails.

Formula: (True Positives + True Negatives) / Total Samples

• Precision: Proportion of predicted phishing emails that are actually phishing.

Formula: True Positives / (True Positives + False Positives)

• Recall (Sensitivity): Proportion of actual phishing emails correctly identified.

Formula: True Positives / (True Positives + False Negatives)

• F1-Score: Harmonic mean of precision and recall, balancing the two.

Formula: 2 (Precision Recall) / (Precision + Recall)

Step 12: Confusion matrix visualization

# Confusion matrix visualization

conf_matrix = confusion_matrix(y_test, y_pred)
sns.heatmap(conf_matrix, annot=True, fmt='d', cmap='Blues', xticklabels=['Safe', 'Phishing'], yticklabels=['Safe', 'Phishing'])
plt.title('Confusion Matrix')
plt.xlabel('Predicted')
plt.ylabel('Actual')
plt.show()

Output:

Visualizing Results: A heatmap of the confusion matrix visually represents true positives, false positives, true negatives, and false negatives in model predictions.

The image shows a confusion matrix for a phishing email detection model. A confusion matrix is a performance measurement tool used in classification problems to evaluate how well the model predicts each class.

Key Elements in the Confusion Matrix:

1. Rows (Actual): Represent the true class labels (ground truth).

• Safe: Emails that are not phishing (legitimate).

• Phishing: Emails that are phishing (malicious).

2. Columns (Predicted): Represent the predicted class labels made by the model.

• Safe: Emails classified by the model as legitimate.

• Phishing: Emails classified by the model as phishing.

3. Four Quadrants:

• Top-Left (True Negatives — TN):

• Value: 2194

• Represents the number of legitimate emails correctly classified as “Safe.”

• Top-Right (False Positives — FP):

• Value: 79

• Represents the number of legitimate emails incorrectly classified as “Phishing.”

• Bottom-Left (False Negatives — FN):

• Value: 62

• Represents the number of phishing emails incorrectly classified as “Safe.”

• Bottom-Right (True Positives — TP):

• Value: 1395

• Represents the number of phishing emails correctly classified as “Phishing.”

Interpretation of Values:

• True Positives (1395): The model identified 1395 phishing emails as phishing.

• True Negatives (2194): The model correctly recognized 2194 legitimate emails as safe.

• False Positives (79): The model mistakenly flagged 79 legitimate emails as phishing.

• False Negatives (62): The model failed to detect 62 phishing emails and classified them as legitimate.

Metrics Derived from the Confusion Matrix:

1. Accuracy: Measures the overall correctness of the model.

[embed]

[embed]

2. Precision (for phishing detection): Measures how many emails predicted as phishing are truly phishing.

[embed]

[embed]

3. Recall (Sensitivity): Measures how many actual phishing emails are detected.

[embed]

[embed]

4. F1-Score: Harmonic mean of precision and recall, balancing false positives and false negatives.

[embed]

[embed]

Observations:

• The model performs well, achieving a high accuracy (96.2%) and recall (95.7%), which is critical for phishing detection.

• The false positives (79) and false negatives (62) are relatively low, but the false negatives are more concerning as they represent undetected phishing emails, which could lead to security breaches.

• Precision (94.6%) indicates the model is effective at minimizing false alarms but may still occasionally misclassify legitimate emails.

Step 13: Save the model and vectorizer for deployment

import joblib
joblib.dump(model, 'phishing_email_model.pkl')
joblib.dump(tfidf, 'tfidf_vectorizer.pkl')

Output:

['tfidf_vectorizer.pkl']

Model Persistence: The trained model and TF-IDF vectorizer are saved to disk using Joblib for future use without needing to retrain them each time.

Conclusion

This project demonstrates how students can leverage open-source technologies to build practical AI solutions for pressing cybersecurity challenges. With a focus on minimal resources, this initiative highlights innovation and resourcefulness in the field of phishing detection.