Improving F1 Score for your Machine Learning Projects

The F1 score is a metric for evaluating machine learning models. It combines particular measures into a single value that’s more useful.

I encountered this metric at a hackathon, which was used as a measure of how effectively a model predicted values. The project was a financial inclusion project that looked into 3 countries in Southern Africa.

The model would help financial institutions use variables that give a financial index score. The financial index score would help them find businesses that would best fit their various products. It would also help identify businesses with the least volatility when providing loan facilities.

The biggest challenges with the dataset were cleaning and feature engineering, and overcoming its imbalanced nature. The imbalanced dataset is why the F1 score was used to measure the models’ performance in the hackathon.

What is F1 Score?

The F1 score is a commonly used metric for evaluating the performance of a classification model. It’s calculated by combining recall and precision into a single value.

F1 = 2 x (Precision x Recall) / (precision + Recall)

Recall measures the coverage of actual positive cases, while precision measures the accuracy of positive predictions. Combining them yields a better mean, giving higher weight to lower values. Therefore, if the precision or recall is low, the F1 score drops.

The metric is commonly used in classification models. Classification models are supervised machine learning algorithms used to predict discrete categorical labels.

Some classification models in this project included Random Forest (best performer), XGBoost, and LightGBM.

This guide helps those who want to improve the F1 score of prediction models that handle imbalanced data.

Tactics to Improve Your F1 Score in Imbalanced Datasets

1. Data Cleaning and Feature Engineering

[embed]

Data cleaning is exactly as the phrase means. It involves detecting, correcting, or deleting poorly formatted data from a dataset. Best practices include:

• Filling up the nulls with either mean, median, or mode. They are measures of central tendency that provide an unbiased method for filling in null values. Remember to handle outliers, which may skew the measures of central tendency.
• Also, remember to remove redundant columns that can cause multicollinearity or leakage. This means first drop the target variable, i.e., what you are predicting.
• After doing feature engineering, also drop some of those variables to minimise the noise during modelling. However, do this with caution; you can use the important features from your models or the correlation matrix to find the unwanted features.
• Finally, data can also contain text. Remember, Python is case or character sensitive,so ensure you handle the text data as well. Don’t fear to summarize redundant values into manageable summaries, for example:

def clean_survey_column(series: pd.Series) -> pd.Series:
    """
    Clean survey-style categorical responses into 3 buckets:
    - 'Have now'
    - 'Dont have'
    - 'Unknown'
    """
    mapping = {
        "Have now": "Have now",
        "Never had": "Dont have",
        "Used to have but don't have now": "Dont have",
        "Used to have but don’t have now": "Dont have",  # handle both apostrophe styles
        "Don't know": "Unknown",
        "Don’t know (Do not show)": "Unknown"
    }
    return series.map(mapping).fillna("Unknown")

# list columns to clean
cols_to_clean = [
    "medical_insurance",
    "funeral_insurance",
    "has_credit_card",
    "has_debit_card",
]

# Apply cleaning function to each column
for col in cols_to_clean:
    train_fe[col + "_clean"] = clean_survey_column(train_df[col])

The end result is:
medical_insuarance_clean
Dont have    1300
Unknown       996
Have now      109
Name: count, dtype: int64

Feature engineering is the process of transforming raw data into informative features.

The caveat, however, is that you need domain knowledge or be a subject matter expert. But with artificial intelligence, search engines, and social media, you have access to domain knowledge at your fingertips.

That said, remember to share data in accordance with data privacy laws, especially with AI, which may use the input as part of its training data, thereby breaching those laws.

A better strategy when using AI for your project is to share only your variable names and their definitions, along with the debugging issues. With variable names, the AI can give you proper feature engineering methods without revealing sensitive information.

Data cleaning and feature engineering lay the foundation for your model’s effectiveness and, consequently, a high F1 score.

2. Resampling Techniques

My model performed well after cleaning and feature engineering, but precision was low due to the imbalanced dataset. A strategy that a friend shared with me was resampling methods.

Resampling methods are techniques for generating new data points from an existing dataset.

For example, the Financial Health Index in the project had 3 parameters: “High, Low, and Medium”. In my project, the financial health index of most companies (about 70%) was low, meaning the prediction would lean towards this, blocking out many deserving businesses from financial support.

Resampling is an important process in imbalanced data, which can be addressed in two main ways: undersampling or oversampling.

Undersampling reduces the majority class instances to balance the dataset. Its Commonly used for large datasets.

On the other hand, we have oversampling, which creates synthetic samples of the minority class.

SMOTE, or Synthetic Minority over-sampling technique, is an ML algorithm that creates synthetic samples rather than duplicating them.

In my case, Smote-Tomek worked best: SMOTE oversamples, then Tomek removes overlapping instances to balance and clean the classes. The algorithm improved my F1 score by 60%.

Read more about other SMOTE methods.

3. Choose the Right Model

You have to pick the right model for each problem. In my case, I tried four different models to find the one that gives the best prediction.

Random forest worked best in my situation. The reason is that it balances class distribution by undersampling the majority class, therefore boosting the effects of the Smote-Tomek oversampling the minority class.

Notably, the solutions didn’t get me to the top of the leaderboard. However, the solution that clinched the top positions leveraged a 4-model ensemble, hence producing a higher F1 score.

In my case, an example of using an ensemble model is not using my models separately but letting them complement each other.

Random forest for balancing class weights, XGboost for minority class recall, and Logistic regression for an interpretable benchmark. Topics to explore for a proper ensemble model include stacking, voting, and weighted average in machine learning.