Testing ML Models for Classification of Flight Delays

Part 5 in Predicting Flight Delays

The data has been scraped, explored, and cleaned so now it is finally time to test out some machine learning models! All code can be found on my github https://github.com/dlosowyj/flight-delay-forecasting.

Mt Hood from PDX.

Overview of Models and Testing Plan

There are a myriad of classification models to choose from spanning simple linear classifiers like logistic regressors to ensemble models like random forests. I will not be able to test all of them, but I will sample a range of models, attempting to strike a balance between model complexity and model accuracy. The models I plan to test are:

1. Logistic Regression
2. Nonlinear SVM
3. Random Forest
4. Neural Network.

Models will go through hyperparameter optimization, when possible, to give each one the best chance at properly fitting the data and they will be graded through accuracy, precision, recall, and f1 scores. The data will still be the flight delay data from PDX from January 2021 through September 2025.

Train-Test Split

Fortunately, there is no huge asymmetry to the data in terms of the target variable, so I can work with a relatively straightforward split. My only consideration is to restrict the training data to the first ~3.75 years of data so that the final year of data can be used for testing with no data leakage. This results in a roughly 75%-25% split in the training to test data.

cutoff_date = delay_df['Date (MM/DD/YYYY)'].max() - pd.offsets.DateOffset(years=1)
train = delay_df[delay_df['Date (MM/DD/YYYY)'] < cutoff_date]
test = delay_df[delay_df['Date (MM/DD/YYYY)'] >= cutoff_date]

X_train = train.drop(['Delayed', 'Date (MM/DD/YYYY)'], axis=1)
y_train = train['Delayed']
X_test = test.drop(['Delayed', 'Date (MM/DD/YYYY)'], axis=1)
y_test = test['Delayed']

print(len(X_train) / (len(X_train) + len(X_test)))from sklearn.model_selection import train_test_split

features_df = delay_df.drop(['Delayed'], axis=1)
labels_df = delay_df['Delayed']
X_train, X_test, y_train, y_test = train_test_split(features_df, labels_df, test_size=0.2)

Logistic Regression

Logistic regression is, in many ways, the fundamental classifier. It is easily interpreted in terms of a probability, but can also just be used as a hard classifier across a decision boundary. Here, I have chosen to use the SGDClassifier with the loss function being log_loss so that it will function just like logistic regression would. The advantage of using the SGDClassifier is that it can be parallelized through the n_jobs parameter, which is not possible in scikit-learn’s standard LogisticRegression implementation. I am also making use of RandomizedSearchCV to choose the optimal parameters for the multiplier of the regularization alpha and the elastic net mixing parameter l1_ratio.

from scipy.stats import loguniform, uniform
from sklearn.linear_model import SGDClassifier
from sklearn.model_selection import RandomizedSearchCV

logistic_pipeline = Pipeline([
    ('data_transformer', feature_transformer),
    ('logistic_clf', SGDClassifier(loss='log_loss', penalty='elasticnet', n_jobs=-1)) # log_loss makes it a logistic regression
])

logistic_param_distribs = {'logistic_clf__alpha': loguniform(1e-6, 1),
                           'logistic_clf__l1_ratio': uniform(loc=0., scale=1.)}
logistic_search = RandomizedSearchCV(logistic_pipeline, param_distributions=logistic_param_distribs, 
                                n_iter=100, cv=3, n_jobs=-1, scoring='balanced_accuracy')
logistic_search.fit(X_train, y_train.values.ravel())

Examining the accuracy, precision, recall, and f1 score of the best logistic regression model, it has a middling degree of success. Its accuracy is above 50% so better than pure guessing (although worse than purely guessing on-time), but it does seem to struggle more with precision suggesting that it is “overeager” in identifying more delayed flights than there actually are. Before jumping to additional conclusions about this being an unsuitable model, I will investigate other models for comparison.

from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score

best_logistic_model = logistic_search.best_estimator_
logistic_pred = best_logistic_model.predict(X_test)
logistic_results = {'Accuracy': accuracy_score(y_test, logistic_pred),
                    'Precision': precision_score(y_test, logistic_pred),
                    'Recall': recall_score(y_test, logistic_pred),
                    'F1': f1_score(y_test, logistic_pred)}
print(logistic_results)

{'Accuracy': 0.5967421246321696, 
 'Precision': 0.5023751125461827, 
 'Recall': 0.5770581836985789, 
 'F1': 0.5371330882413962}

Nonlinear SVM

Support vector machines (SVMs) are famously flexible models, particularly when combined with the appropriate kernel. With my assumptions that delayed flights will share similar characteristics and be in close proximity in the feature space, a radial basis function (RBF) should give a good measure of similarity through Euclidean distance. The categorical features have gone through one hot encoding while the numerical features have been transformed via sine and cosine meaning that all features are already normalized appropriately. I will tune the regularization parameter C, again, through RandomizedSearchCV along with gamma, which sets the variance (or effectively the influence) of individual points. (I did also set the max_iter to 2000 because this model seemed to struggle converging for this dataset.)

from sklearn.svm import SVC

svm_pipeline = Pipeline([
    ('data_transformer', feature_transformer),
    ('svm_clf', SVC(kernel='rbf', max_iter=2000)) # radial basis function kernel
])

svm_param_distribs = {'svm_clf__C': uniform(loc=0., scale=1.),
                      'svm_clf__gamma': loguniform(1e-6, 1)}
svm_search = RandomizedSearchCV(svm_pipeline, param_distributions=svm_param_distribs, 
                                n_iter=30, cv=3, n_jobs=-1, scoring='balanced_accuracy')
svm_search.fit(X_train, y_train.values.ravel())

Interestingly, this model fares slightly worse than the logistic regression model with an accuracy of 57%. This is a larger dataset so my restriction on the number of iterations may have hampered the SVM from finding an appropriate fit although forays into larger numbers of iterations did not show significant improvement. Moreover, the balance between the categorical and numerical features could also be causing some issues. Since there are so many options for flights in terms of carrier, tail number, and destination airport, there is likely always a distance contribution of 3*sqrt(2) to the RBF kernel. This can hamper the model from finding similarities between delayed and on-time flights.

best_svm_model = svm_search.best_estimator_
svm_pred = best_svm_model.predict(X_test)
svm_results = {'Accuracy': accuracy_score(y_test, svm_pred),
                    'Precision': precision_score(y_test, svm_pred),
                    'Recall': recall_score(y_test, svm_pred),
                    'F1': f1_score(y_test, svm_pred)}
print(svm_results)

{'Accuracy': 0.5799900224855579, 
 'Precision': 0.4833291884274227, 
 'Recall': 0.5198195467270555, 
 'F1': 0.5009106842159524}

Random Forest

The next model I am testing is a transition away from single classifiers to an ensemble model. I am hoping that a random forest composed of many separate trees will be able to capture the odd, multi-dimensional complexity that this dataset has. Another benefit of the random forest is that I will be able to investigate which features are the most important for classification, which could help me refine my models in the future. For this model, there are a large number of parameters to tune, most of them centered on the properties of the trees including the maximum depth, maximum number of leaves, etc.

from sklearn.ensemble import RandomForestClassifier
from scipy.stats import randint
from sklearn.model_selection import RandomizedSearchCV

rnd_forest_pipeline = Pipeline([
    ('data_transformer', feature_transformer),
    ('rnd_forest_clf', RandomForestClassifier(criterion='gini', n_jobs=-1))
])

# We will be tuning our decision tree
param_distribs = {'rnd_forest_clf__min_samples_split': randint(low=10, high=30),
                  'rnd_forest_clf__min_samples_leaf': randint(low=10, high=30),
                  'rnd_forest_clf__max_leaf_nodes': randint(low=100, high=300),
                  'rnd_forest_clf__max_depth': randint(low=10, high=30),
                  'rnd_forest_clf__max_features': randint(low=100, high=300)}
rnd_forest_search = RandomizedSearchCV(rnd_forest_pipeline, param_distributions=param_distribs, 
                                n_iter=60, cv=3, scoring='balanced_accuracy')
rnd_forest_search.fit(X_train, y_train.values.ravel())

While technically the best model so far at 68% accuracy, this model is still not very accurate and particularly struggles with recall — it is having difficulty identifying all of the delayed flights.

best_forest_model = rnd_forest_search.best_estimator_
best_forest_model = train_forest()
forest_pred = best_forest_model.predict(X_test)
forest_results = {'Accuracy': accuracy_score(y_test, forest_pred),
               'Precision': precision_score(y_test, forest_pred),
               'Recall': recall_score(y_test, forest_pred),
               'F1': f1_score(y_test, forest_pred)}
print(forest_results)

{'Accuracy': 0.6704528200938464, 
 'Precision': 0.6088344423025102, 
 'Recall': 0.5237424439649793, 
 'F1': 0.5630919059852}

If I take a look at the random forest’s most important features, I do get some insight into what is driving this model and, possibly, what is limiting the other models’ accuracies. The dates and times do all appear in the top fifteen, but then several carrier codes are dominating the list, particularly Southwest airlines along with a few destination airports. The random forest model, at least, seems to find great predictive power from the specific airline carrying a flight. With that much stake being placed on thecarrier, it is possible that the model is overwhelmed by the carrier so that the date and time (aside from the sine transformation of time) just cannot impact the prediction significantly.

feature_names = best_forest_model.named_steps['data_transformer'].get_feature_names_out(input_features=X_train.columns)
rnd_forest_importances = best_forest_model.named_steps['rnd_forest_clf'].feature_importances_
feature_importances_series = pd.Series(rnd_forest_importances, index=feature_names).sort_values(ascending=False)
print(feature_importances_series[:15])

categorical__Carrier Code_WN            0.281515
hour_sin__Hour_sin                      0.200593
categorical__Carrier Code_AS            0.093664
hour_cos__Hour_cos                      0.064638
categorical__Carrier Code_OO            0.059814
categorical__Carrier Code_QX            0.057196
categorical__Destination Airport_SEA    0.044340
categorical__Carrier Code_DL            0.028912
day_cos__DayOfYear_cos                  0.022718
categorical__Destination Airport_MDW    0.009551
categorical__Destination Airport_LGA    0.009378
day_sin__DayOfYear_sin                  0.008862
categorical__Destination Airport_GEG    0.007694
categorical__Destination Airport_BOI    0.006444
categorical__Carrier Code_AA            0.005622

Neural Network

Training a neural network will take me outside of scikit-learn’s offerings. I will be using PyTorch to create a network with two hidden layers of 128 and then 64 nodes, which I hope will be able to capture some of the complexity of the wide feature range and then pare it down to a usable level. Since there are a lot of very sparsely populated features in this dataset, I am also using LeakyReLU to make sure that nodes are not dying and that gradients are not vanishing prematurely just because a feature has not been activated sufficiently recently. Finally, I will be using AdamW as the optimizer to prevent possible overfitting and ExponentialLR to smoothly step down the learning rate.

import torch.nn as nn
import torch.optim as optim
from torch.optim.lr_scheduler import ExponentialLR

class BinaryClassifier(nn.Module):
    def __init__(self, input_dim):
        super(BinaryClassifier, self).__init__()
        self.layer_1 = nn.Linear(input_dim, 128)
        self.bn = nn.BatchNorm1d(128)    #re-center the sum of features to prevent predicting 0 every time
        self.activation = nn.LeakyReLU(0.01)
        self.dropout = nn.Dropout(p=0.5)
        self.layer_2 = nn.Linear(128, 64)
        self.layer_out = nn.Linear(64, 1) # 1 output unit for binary classification

    def forward(self, x):
        x = self.layer_1(x)
        x = self.bn(x)
        x = self.activation(x)
        x = self.dropout(x)
        x = self.layer_2(x)
        x = self.activation(x)
        x = self.dropout(x)
        x = self.layer_out(x)
        return x

model = BinaryClassifier(input_dim=X_train_nn.shape[1])

criterion = nn.BCEWithLogitsLoss()
optimizer = optim.AdamW(model.parameters(), lr=1e-5, weight_decay=1e-2)
scheduler = ExponentialLR(optimizer, gamma=0.98)

train_losses = [] # Storage for plotting
val_losses = []   # Storage for plotting
epochs = 15

for epoch in range(epochs):
    model.train()
    batch_train_losses = []

    for inputs, labels in train_loader:
        inputs = inputs.to(device).float()
        labels = labels.to(device).float().view(-1, 1) # SHAPE GUARD

        optimizer.zero_grad()
        outputs = model(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()
        batch_train_losses.append(loss.item())

    # --- Validation ---
    model.eval()
    batch_val_losses = []
    all_preds, all_labels = [], []

    with torch.no_grad():
        for inputs, labels in val_loader:
            inputs = inputs.to(device).float()
            labels = labels.to(device).float().view(-1, 1)
            logits = model(inputs)
            loss = criterion(logits, labels)
            batch_val_losses.append(loss.item())

            probs = torch.sigmoid(logits)
            preds = (probs > 0.5).float()
            all_preds.extend(preds.cpu().numpy())
            all_labels.extend(labels.cpu().numpy())

    scheduler.step()

    # Store average losses for this epoch
    epoch_train_loss = np.mean(batch_train_losses)
    epoch_val_loss = np.mean(batch_val_losses)
    train_losses.append(epoch_train_loss)
    val_losses.append(epoch_val_loss)

    print(f"Epoch {epoch+1}/{epochs} | Train Loss: {epoch_train_loss:.4f} | Val Loss: {epoch_val_loss:.4f}")

Looking at the neural network’s performance, it does have similar accuracy to the random forest at 67%, but actually exceeds it in the recall metric (57% for the neural net compared to 52% for the random forest). If nothing else, it suggests that I can trust a neural network above a random forest to identify a larger proportion of delayed flights.

all_labels = []
all_preds = []

# Ensure model is on the correct device (CPU or GPU)
model.eval() 
with torch.no_grad():
    for inputs, labels in test_loader:
        inputs, labels = inputs.to(device), labels.to(device)

        outputs = model(inputs)
        # _, preds = torch.max(outputs, 1) # Get the predicted class index
        probs = torch.sigmoid(outputs)
        preds = (probs > 0.5).float()

        # Move to CPU and convert to NumPy for sklearn compatibility
        all_labels.extend(labels.cpu().numpy())
        all_preds.extend(preds.cpu().numpy())

nn_results = {'Accuracy': accuracy_score(all_labels, all_preds),
              'Precision': precision_score(all_labels, all_preds),
              'Recall': recall_score(all_labels, all_preds),
              'F1': f1_score(all_labels, all_preds)}
print(nn_results)

{'Accuracy': 0.674761949519561, 
 'Precision': 0.6042108821816953, 
 'Recall': 0.5736345642909363, 
 'F1': 0.5885258497676631}

Summary

Going through the gamut of models, there are two clear groupings: the logistic regression and nonlinear SVM models in one, then random forest and neural network models in the other. Unfortunately, not even the latter group had truly outstanding performance. Both the random forest and neural network functioned with 67% accuracy, but were prone to missing delayed flights in classification through their depressed recall metrics. In a practical case, that would mean waiting at the gate for a flight that the model was not able to classify as being delayed. And while the neural network did perform slightly better than the random forest model, the utility of viewing the most important features of the random forest model would make it my choice in flight delay data.

With the models compared, it is finally time to expand my fits to more airports outside of PDX. The increased data may shed some light on the additional importance of, say, day of the year when dealing with airports in chillier climes. Given the memory constraints on my PC, however, including more airports will take some finagling so that will be broached in the next part of this project.

Other Flight Delay Posts

Part 1: Predicting Flight Delays Intro.

Part 2: Web Scraping Flight Delay Data

Part 3: Exploring Flight Delay Data

Part 4: Data Cleaning for Classification of Flight Delays

Part 6: A Random Forest Model for Flight Delay Classification