Analyzing and Predicting Airline Delays: A Comprehensive Data Science Approach

Photo by Joshua Sukoff on Unsplash

Introduction

Air travel is one of the most time-sensitive forms of transportation, and delays can cause frustration for both airlines and passengers. In this analysis, I used the **Airline On-Time Performance Dataset** provided by the U.S. Department of Transportation’s Bureau of Transportation Statistics (BTS) to explore patterns in delays, cancellations, and diversions. Using various machine learning models and visualizations, I aimed to identify trends, uncover the primary causes of delays, and provide predictive insights.

Understanding the Datset

The dataset spans from January 2024 to July 2024, containing over 4 million records and 110 columns detailing flight operations, including departure and arrival times, delays, cancellations, taxi times, and more.

Key Variables:

• FlightDate: Date of flight
• DepDelay, ArrDelay: Departure and Arrival delays in minutes
• Cancelled, Diverted: Indicators for flight cancellations and diversions
• Distance: Distance between origin and destination airports

The first step was data preprocessing to clean the dataset, handle missing values, and consolidate the monthly datasets into a comprehensive dataset of 4,095,932 flights.

Data Preprocessing

A significant part of the process was ensuring data quality for machine learning models. I performed the following steps:

• Null value handling: Columns with missing or incorrect values were cleaned. For example, 77% of the flights had missing carrier delay information, likely because these flights were not delayed by airlines.
• Time Conversion: The flight times were initially in HHMM format. I converted these into a readable 12-hour AM/PM format using a custom function, enhancing clarity for the analysis.

GitHub Link to access the ipynb file

Key Visualizations and Analysis

1. Flight Delays Across Airlines

This bar chart showcases the number of flights operated by each airline from January to July 2024. Southwest Airlines (WN) operated the highest number of flights, followed by Delta (DL) and American Airlines (AA). The visual highlights the operational scale of these airlines, with the largest players contributing significantly to total air traffic. Smaller airlines like Spirit (NK) and Hawaiian (HA) operate fewer flights in comparison.

2. Determining the Status of the Flight

In this visualization, the flight statuses are divided into four categories: on-time, delayed, canceled, and diverted. A large portion of flights were on time, but a noticeable percentage experienced delays. Cancellations were less frequent, while diversions formed a small segment.

3. Average Departure and Arrival Delay Over Time & Cancellation Rate

The line graph illustrates the average departure and arrival delays from January to July 2024. Notably, January and July displayed the highest delays, possibly due to adverse weather conditions and the surge in summer travel, respectively. Cancellation rates also spiked during these months, reflecting the correlation between high delay periods and cancellations.

4. Distribution of Delay Across Different Airlines

This bar chart breaks down delay percentages across different airlines. While major airlines like American (AA) and Frontier (F9) report higher numbers of delayed flights (due to their volume of operations), even smaller airlines like Spirit (NK) showed moderate delay percentages.

5. Highest Average Departure and Arrival Delays by Top 20 Airports

This visualization highlights the top 20 airports with the highest average delays. Airports like John F. Kennedy (JFK) has reported highest delay of almost 725+ minutes in Departure as well as Arrival delays.

6. Average Delay by Cause

In this visualization, we break down delays into different causes: Carrier, Weather, Security, Late-arriving aircraft, and National Aviation System (NAS) delays. Carrier-related issues and NAS delays were the most frequent causes of delay with almost 75%, while security delays were relatively rare.

7. Departure and Arrival Delays by Flight Distance

This visualization explores the relationship between flight distance and delays. Short and Medium-distance (0–3,000 miles) were observed to have the highest percentage of delays, while long-haul flights tended to experience fewer delays.

8. Diverted Flights by Airlines and Airports

This analysis visualizes the diversion rates by airline and airport. Southwest (WN) and American (AA) had higher diversion rates, particularly at congested airport like Dallas Fort Worth (DFW).

9. Correlation Heat-Map of Flight Distance

The heat-map analysis reveals that departure delays (DepDelay) and arrival delays (ArrDelay) are highly correlated with a value of 0.95. This strong correlation indicates that once a flight is delayed at departure, it is almost certain to experience delays upon arrival. Additionally, departure and arrival delays also have moderate correlation with carrier delay with value of 0.63 and 0.60. Scatter plot below can be viewed to check the correlation.

10. Common Reasons for Flight Cancellations

This chart breaks down the primary causes of flight cancellations. Weather (B) emerges as the top reason for cancellations, followed by carrier-related (A) issues. Understanding these cancellation causes is crucial for airlines to implement better contingency plans, such as backup resources or improved communication systems, to handle adverse weather conditions or technical challenges more efficiently.

Predicting Delays with Machine Learning

In the final section of our project, we applied several machine learning models to predict flight delays. These models were chosen to compare their performance on various metrics and identify the best model for accurate delay prediction. The models include Linear Regression, Decision Tree, Random Forest, and XGBoost.

Linear Regression Model

Linear regression performed quite well in predicting delays, showing strong correlation between the input features and target variable. The R² score was particularly high, averaging 0.95, which indicates that about 95% of the variance in the delay data was explained by the model. The average RMSE was 14.20, indicating a reasonable margin of error for predictions.

Decision Tree Model

The Decision Tree model, while interpretable, didn’t perform as well as Linear Regression. The average RMSE was 21.24, and the R² score averaged 0.89, indicating lower predictive power. The higher error rates suggest that this model might be overfitting or unable to capture complex relationships as well as other models.

Random Forest Model

Random Forest showed similar performance to Linear Regression, with an average R² of 0.94 and RMSE of 14.93. The model captured more complex patterns compared to the Decision Tree while avoiding overfitting, thanks to ensemble techniques. Random Forest proved to be a robust option for this predictive task.

XGBoost Model

XGBoost, while powerful, didn’t outperform Random Forest in this scenario. With an average R² of 0.92 and RMSE of 18.09, XGBoost had relatively higher error values compared to Random Forest. This could be due to the complexity of the data or the need for further tuning.

Conclusion

The Airline Delay Analysis and Prediction Project provided a comprehensive understanding of the factors contributing to flight delays. By analyzing data from January to July 2024, we identified key trends and built machine learning models capable of predicting delays with significant accuracy. Insights from these visualizations and models can help airlines and airports optimize their operations, reduce delays, and enhance the overall efficiency of the air transportation system.

Among the models tested, Linear Regression and Random Forest emerged as the most accurate predictors for airline delays. Both had high R² scores (~0.95 for Linear Regression and ~0.94 for Random Forest), indicating strong predictive power. Random Forest, in particular, stands out due to its ability to generalize well across the data without overfitting, making it the recommended model for deployment in this context.

In contrast, Decision Tree and XGBoost showed slightly higher errors and lower R² scores, suggesting they might not capture the complexity of the dataset as well as the other models.