Heart Failure Prediction with PySpark and Logistic Regression

In this tutorial, we will build a Logistic Regression model using PySpark on Apache Zeppelin. The data will be stored in HDFS, and we will use Zeppelin as an interactive environment to write and execute PySpark code. This approach enables efficient data storage, scalable processing, and seamless visualization of results.

HDFS (Hadoop Distributed File System) is a distributed file system for storing large datasets across multiple computers.

PySpark is a Python library for Apache Spark, used to process big data in a distributed manner.

Zeppelin is a web-based notebook that supports writing and running Spark, SQL, and Python code with interactive visualization.

First, let’s understand how Logistic Regression works.

What is Logistic Regression?

Logistic Regression is a classification algorithm used to model the probability of a categorical outcome. It applies a sigmoid function to map input features to values between 0 and 1, making it suitable for binary and multi-class classification problems.

Why is Logistics Regression Important?

Logistic regression is widely used in artificial intelligence (AI) and machine learning (ML) for classification tasks and predictive analytics. It helps organizations make data-driven decisions, improving efficiency, reducing operational costs, and enabling scalable solutions. Some common applications include:

• Business: Predicting employee retention rates.
• Healthcare: Assessing disease risks based on patient history.
• Finance: Detecting fraudulent transactions and credit risk assessment.

Advantages of Logistic Regression

1. Simplicity: Easy to use without requiring deep ML expertise.
2. Speed: Handles large datasets with minimal computational resources.
3. Flexibility: Works for binary and multi-class classification, and integrates well with other ML techniques.
4. Visualization: Provides clear insights into decision-making, making debugging easier.

How Logistic Regression Works

Logistic regression is a statistical model that uses the logistic function, or logit function, in mathematics as the equation between x and y. The logit function maps y as a sigmoid function of x.

1. Sigmoid Function

The logistic function maps real numbers to a range between 0 and 1:

Source:https://aws.amazon.com/what-is/logistic-regression/

where z is a linear combination of input features.

Source:https://aws.amazon.com/what-is/logistic-regression/

The sigmoid function produces an S-shaped curve, which gradually approaches 0 and 1.

2. Decision Boundary

• If P(Y=1) > 0.5, classify as 1 (Positive)
• If P(Y=1) < 0.5, classify as 0 (Negative)

This threshold can be adjusted depending on the problem.

3. Cost Function (Log Loss)

To measure the model’s performance, we use the Log Loss function:

where y is the actual outcome, and p is the predicted probability.

4. Regularization (Prevents Overfitting)

• L1 Regularization (Lasso): Encourages sparsity by forcing some coefficients to be exactly zero, effectively performing feature selection.
• L2 Regularization (Ridge): Penalizes large coefficients, reducing variance and preventing overfitting.
• Elastic Net: A combination of L1 and L2, balancing feature selection and regularization.

Types of logistic regression analysis

1. Binary Logistic Regression: Used for two possible outcomes (e.g., Yes/No, 0/1).
2. Multinomial Logistic Regression: Handles multiple unordered categories (e.g., price changes of 25%, 50%, etc.).
3. Ordinal Logistic Regression: Used for ranked outcomes (e.g., Poor, Fair, Good, Excellent).

Hands-on: Implementing Logistic Regression with PySpark on AWS EMR

Now that we understand the theory behind Logistic Regression, let’s put it into practice. In this section, we will implement a Heart Disease Prediction Model using PySpark on AWS EMR (Elastic MapReduce).

1. Setting Up the Environment

1.1 Create an EMR Cluster

Navigate to AWS EMR Console → Create Cluster

Select emr-7.4.0

Select Required Applications

• Spark → Enables us to run PySpark for data processing and model training
• Hadoop → Provides HDFS for storing our dataset
• Zeppelin → A web-based notebook for writing and executing PySpark code

While waiting for the EMR Cluster to be ready, we need to configure EC2 Security Group to allow connections to the primary node. This will enable us to access Zeppelin, Hadoop, and Spark services.

Once connected via SSH, you should see the Amazon Linux 2023 shell with the EMR banner.

2. Loading the Dataset

The dataset used in this project is the Heart Failure Prediction Dataset, originally from **Kaggle**. To simplify access, we have uploaded the dataset to Dropbox for direct downloading.

2.1 Download the Data

• Uses wget to fetch the dataset from Dropbox.

wget "https://www.dropbox.com/scl/fi/i0h9xkozw4vtecwovrp0h/heart.csv?rlkey=q1sknkhcuk58qz0658ogi4k2w&st=4au0b602&dl=1" -O /home/hadoop/heart.csv

2.2 Store Data in HDFS

Before we can use the dataset in our Hadoop Distributed File System (HDFS), we need to create a directory to store it. Run the following command:

#Create a Folder in HDFS for the Dataset
hadoop fs -mkdir -p /user/hadoop/heart_failure

#Copy the Dataset to HDFS
hadoop fs -put /home/hadoop/heart.csv /user/hadoop/heart_failure/

#Check whether the file has been successfully uploaded
hadoop fs -ls /user/hadoop/heart_failure/

Alternatively, you can browse the HDFS file system via the Hadoop Web UI

This ensures that the dataset is stored correctly in HDFS and ready for further processing in PySpark

3. Configuring Zeppelin for User Authentication and Spark Interpreter Settings

Before running PySpark, we need to set up Zeppelin:

3.1 Enable User Authentication

By default, Zeppelin does not require authentication. To secure access, we will set up authentication so that users must log in.

#Switch to root user
sudo su -

#Navigate to the Zeppelin configuration folder
cd /etc/zeppelin/conf

#Copy the default authentication configuration file
cp shiro.ini.template shiro.ini

#Edit the shiro.ini file
vi shiro.ini

Inside the [users] section, add the following:

hadoop = password, admin

This creates a user hadoop with the password password and admin privileges.

Restart Zeppelin to apply changes:

systemctl restart zeppelin

3.2 Configure Spark Interpreter

To ensure that Zeppelin executes Spark jobs correctly under the hadoop user, we need to update the Spark Interpreter settings.

Go to Zeppelin’s web interface and log in with

• Username: hadoop
• Password: password
• Go to Interpreter → Edit Spark.

Modify the interpreter settings:

• Set the interpreter to run per user: The interpreter will be instantiated Per User in isolated process
• Enable user impersonation: Check "User Impersonate"
• Click Save to apply changes.

3.3 Install Required Libraries (numpy)

Since PySpark may require external dependencies like numpy, install it on the EMR cluster:

# Switch to root user
sudo su -

# Install numpy
yum install -y numpy

#Restart Zeppelin again
systemctl restart zeppelin

At this point, Zeppelin is fully configured. You can create a new Notebook and start running PySpark code.

4. Load Data into PySpark

Since we stored our dataset in HDFS, we now load it into Spark.

%pyspark
df = spark.read.csv("heart.csv", header=True, inferSchema=True)
df.show(5)
df.printSchema()  # ดูโครงสร้างข้อมูล
df.describe().show()  # ดูค่าทางสถิติ

Displays the first 5 rows

Shows the structure of the dataset.

Computes basic statistics (mean, std, min, max).

5. Preprocessing

5.1 One-Hot Encoding for Categorical Features We first use StringIndexer to assign numeric labels to categorical values, then apply One-Hot Encoding to create binary columns.

%pyspark
from pyspark.ml.feature import StringIndexer, OneHotEncoder

# กำหนด Column ที่เป็น Categorical
categorical_cols = ["Sex", "ChestPainType", "RestingECG", "ExerciseAngina", "ST_Slope"]

# การแปลงข้อมูลประเภท Categorical เป็นตัวเลขด้วย StringIndexer
indexers = [StringIndexer(inputCol=col, outputCol=col+"_index").fit(df) for col in categorical_cols]

for indexer in indexers:
    df = indexer.transform(df)

# ทำ One-Hot Encoding
encoder = OneHotEncoder(inputCols=[col+"_index" for col in categorical_cols],
                        outputCols=[col+"_encoded" for col in categorical_cols])

df = encoder.fit(df).transform(df)

df.show(5)

• StringIndexer assigns numerical values to categorical variables.
• OneHotEncoder ensures categorical values are represented as separate binary columns.

5.2 Combine Features into a Single Vector (VectorAssembler)

Machine learning models in PySpark require features to be in a single vector column rather than separate columns.

%pyspark
from pyspark.ml.feature import VectorAssembler

# เลือกเฉพาะฟีเจอร์ตัวเลขและฟีเจอร์ที่ถูก One-Hot Encoding
numeric_features = ['Age', 'RestingBP', 'Cholesterol', 'FastingBS', 'MaxHR', 'Oldpeak']
encoded_features = [col+"_encoded" for col in categorical_cols]

assembler = VectorAssembler(inputCols=numeric_features + encoded_features, outputCol="features")
df = assembler.transform(df)
df = df.select("features", "HeartDisease")

df.show(5)

• VectorAssembler consolidates all numerical and categorical features into a single column named "features".
• This transformation is required for Spark ML models.

6. Splitting the Data (Train/Test)

To evaluate model performance, we split the dataset into training (80%) and testing (20%) sets.

%pyspark
train_df, test_df = df.randomSplit([0.8, 0.2], seed=42)
train_df.show(5)

• The training set is used to train the model.
• The test set is used to measure how well the model generalizes to new data.

7. Standard Scaling

Features in different scales (e.g., age vs. cholesterol levels) may cause models to be biased. Standard Scaling normalizes all features to have zero mean and unit variance.

%pyspark
from pyspark.ml.feature import StandardScaler

scaler = StandardScaler(inputCol="features", outputCol="scaled_features", withStd=True, withMean=True)
scaler_model = scaler.fit(train_df)
train_df = scaler_model.transform(train_df)
test_df = scaler_model.transform(test_df)

train_df.show(5)

• Standardization improves model stability and convergence speed.
• The scaler is fitted only on the training data to avoid data leakage.

8. Train Logistic Regression

Logistic Regression is a supervised classification algorithm that models the probability of a binary outcome (Heart Disease = 1 or 0).

%pyspark
from pyspark.ml.classification import LogisticRegression

# สร้างโมเดล Logistic Regression
lr = LogisticRegression(featuresCol="scaled_features", labelCol="HeartDisease",
                        maxIter=500, regParam=0.1, elasticNetParam=0.0, threshold=0.5)
model = lr.fit(train_df)

# ทำนายบน Test Set
predictions = model.transform(test_df)

predictions.select("scaled_features", "prediction", "probability", "HeartDisease").show(5)

• MaxIter = 500 sets the maximum number of iterations for model training. The model will stop early if convergence is achieved before reaching this limit.
• RegParam (0.1) applies regularization to reduce overfitting.
• Threshold = 0.5 means that any probability ≥ 0.5 is classified as Heart Disease = 1.

9. Model Evaluation

Evaluating model performance ensures that our classifier is accurate and reliable.

%pyspark
from pyspark.ml.evaluation import MulticlassClassificationEvaluator, BinaryClassificationEvaluator

# Accuracy
accuracy_evaluator = MulticlassClassificationEvaluator(labelCol="HeartDisease", metricName="accuracy")
accuracy = accuracy_evaluator.evaluate(predictions)

# Precision, Recall, F1-score
precision_evaluator = MulticlassClassificationEvaluator(labelCol="HeartDisease", metricName="weightedPrecision")
recall_evaluator = MulticlassClassificationEvaluator(labelCol="HeartDisease", metricName="weightedRecall")
f1_evaluator = MulticlassClassificationEvaluator(labelCol="HeartDisease", metricName="f1")

precision = precision_evaluator.evaluate(predictions)
recall = recall_evaluator.evaluate(predictions)
f1_score = f1_evaluator.evaluate(predictions)

# ROC-AUC Score
roc_evaluator = BinaryClassificationEvaluator(labelCol="HeartDisease", metricName="areaUnderROC")
roc_auc = roc_evaluator.evaluate(predictions)

# Confusion Matrix
confusion_matrix = predictions.groupBy("prediction", "HeartDisease").count()

# แสดงผล
print(f"===== Logistic Regression Evaluation =====")
print(f"Accuracy: {accuracy:.4f}")
print(f"Precision: {precision:.4f}")
print(f"Recall: {recall:.4f}")
print(f"F1-score: {f1_score:.4f}")
print(f"ROC-AUC Score: {roc_auc:.4f}")
print("\n===== Confusion Matrix =====")
confusion_matrix.show()

• Accuracy: Measures how often the model predicts correctly.
• Precision: Measures correctness of positive predictions.
• Recall: Measures ability to capture positive cases.
• F1-score: Harmonic mean of Precision and Recall.
• ROC-AUC Score: Measures the model’s ability to distinguish between classes.

Model Performance Summary

The test set contained 149 instances after data preprocessing and splitting.

The Logistic Regression model achieved the following results:

• True Positives (TP) = 77 → Correctly predicted as having heart disease.
• True Negatives (TN) = 50 → Correctly predicted as not having heart disease.
• False Positives (FP) = 13 → Incorrectly predicted as having heart disease.
• False Negatives (FN) = 9 → Incorrectly predicted as not having heart disease.

Key evaluation metrics:

• Recall: 85.23% — effectively identifies heart disease cases with relatively low false negatives.
• ROC-AUC Score: 0.9086 — indicates strong ability to differentiate between diseased and non-diseased patients.

However, it is important to note that the test set was relatively small, and the number of correctly and incorrectly predicted instances was limited. This may affect the statistical reliability of the results. Further validation with larger datasets and experimentation with different machine learning models (e.g., Random Forest, Support Vector Machine) are recommended to improve the model’s robustness and minimize potential overfitting.

Conclusion

This project demonstrated how to build a Logistic Regression model using HDFS, PySpark, and Zeppelin for large-scale heart failure prediction.

Key takeaways: — Logistic Regression is effective for binary classification tasks like heart disease detection. — PySpark + HDFS enables scalable and efficient big data processing. — Evaluation metrics (Accuracy, Precision, Recall, ROC-AUC) provide insights into model performance.

This workflow highlights the power of machine learning in a distributed computing environment using Apache Spark.

References / Sources AWS, “What is Logistic Regression?” Available at: AWS Website

This project was conducted by:

*Phantipa Sripradubwong 65xxxxxx208 Akira Sannam 65xxxxxx211 Arisara Saknarai 65xxxxxx695*