ML Model Training with Multiple Features

In the Hello World in ML: Supervised Learning article, I explained how to create a simple machine learning model using a single feature. We trained two models using two popular algorithms: Linear Regression and Decision Tree Regression. The goal of that article was to introduce the fundamentals of machine learning and help us get our feet wet in the world of ML.

In real-world machine learning projects, models are rarely trained using a single feature. Instead, they are trained on multiple features that collectively help the model learn patterns and make predictions. In this article, we will expand on the previous example by adding more features to train our models.

As a machine learning engineer, one of your primary responsibilities is selecting an appropriate learning algorithm and training it on relevant data. Generally, two major factors can lead to poor model performance: a bad algorithm or bad data.

Bad data can take many forms. For example, the dataset may not accurately represent the problem you are trying to solve, or the selected features may have little or no relationship with the target variable. Even when the dataset is large, poor feature selection can significantly reduce a model's effectiveness.

Features

Features are individual characteristics or properties of the data used to train a machine learning model. If your dataset is presented in a tabular format, each column is referred to as a feature, while each row is referred to as an instance (or sample).

In some cases, you may use all available features to train a model and make predictions. However, using every feature does not always produce the best results. Some features may have little or no relationship with the problem you are trying to solve, while others may introduce noise that negatively affects the model’s ability to generalize to unseen data.

Because of this, selecting the right set of features is just as important as collecting a good dataset. There are several feature selection techniques, but in this article, we will rely on domain knowledge. In other words, we will choose features that we already know are likely to influence the target variable (exam_score).

For example, a student’s exam score may depend on factors such as:

1. study_hours
2. class_attendance
3. exam_difficulty

In the previous article, we trained our models using only study_hours. Let’s improve our model by adding another feature:

# Previous line code
features = dataFrame[['study_hours']]

# change it to
features = ['study_hours', 'class_attendance']
X_train = dataFrame[features].copy()

Did you notice something?

I intentionally left out exam_difficulty, even though it was one of the example features listed above. Why?

Machine learning algorithms perform mathematical computations, which means they typically work with numerical values. However, the values in the exam_difficulty column are strings rather than numbers.

How do we handle that?

To answer that question, we need to understand the different types of attributes (features).

Types of Attributes

Features generally fall into two categories:

1. Numerical Attributes
2. Categorical Attributes

Numerical Attributes (Features)

Numerical attributes contain quantitative values that can be used directly in mathematical calculations.

Examples: Study hours, Age, Height, Salary

Categorical Attributes (Features)

Categorical attributes contain a limited number of possible values that represent labels or categories rather than measurable quantities.

Examples: Gender, Course, Country, Difficulty level

Categorical attributes can be divided into three types.

Nominal Attributes: Nominal attributes have no natural ordering or ranking.

Examples:

1. Male, Female
2. Red, Blue, Green
3. Computer Science, Mathematics, Physics

Ordinal Attributes: Ordinal attributes have a meaningful order or ranking

Examples:

1. Easy, Moderate, Hard
2. Poor, Fair, Good, Excellent
3. First, Second, Third

Binary Attributes: with just two possible values (Yes/No)

Examples:

1. Yes / No
2. True / False
3. Pass / Fail

Identifying the Type of exam_difficulty

Let’s inspect the values stored in the exam_difficulty column.

dataFrame['exam_difficulty'].unique()

If you are using the same dataset, the output should contain three values:

['easy', 'moderate', 'hard']

Since these values represent categories and have a natural order of difficulty, we can conclude that exam_difficulty is an ordinal categorical attribute.

The challenge is that machine learning algorithms still require numerical input.

To solve this problem, we use a process called encoding.

Working with Categorical Attributes

Encoding is the process of converting categorical values into numerical values while preserving their meaning.

The encoding technique you choose depends on the type of categorical attribute.

Ordinal Encoding

Ordinal Encoding is used for ordinal categorical attributes because their values already have a meaningful order.

Scikit-Learn provides the OrdinalEncoder class for this purpose.

from sklearn.preprocessing import OrdinalEncoder

# Column
categorical_cols = ['exam_difficulty']

# Selected features
features = ['study_hours', 'exam_difficulty']

# Train Dataset
X_train = dataFrame[features].copy() # Just to keep the original dataset as it's

# Encode
ordinal_encoder = OrdinalEncoder()
encodedCols = ordinal_encoder.fit_transform(X_train[categorical_cols])

# Replace the current values of the attribute with the encoded values
X_train[categorical_cols] = encodedCols

One-Hot Encoding

One-Hot Encoding is used for nominal categorical attributes because there is no meaningful order between their values.

A good example is the course attribute.

Scikit-Learn provides the OneHotEncoder class for this purpose.

from sklearn.preprocessing import OneHotEncoder

# Column
categorical_cols = ['course']

# Selected features
features = ['study_hours', 'course']

# Train Dataset
X_train = dataFrame[features].copy() # Just to keep the original dataset as it's

# Encode
encoder = OneHotEncoder(sparse_output=False)
encodedCols = encoder.fit_transform(X_train[categorical_cols])

# Convert returned Data to DataFrame
encodedDF = pd.DataFrame(encodedCols, 
                         columns=encoder.get_feature_names_out(categorical_cols), 
                         index=X_train.index)

# Replace the current values of the attribute with the encoded values
X_train = X_train.drop(columns=categorical_cols)
X_train = pd.concat([X_train, encodedDF], axis=1);

Unlike Ordinal Encoding, One-Hot Encoding creates a separate column for each possible category.

For example, if a gender column contains only Male and Female, One-Hot Encoding produces two new columns:

Only one column can have a value of 1 for a given row, while the others remain 0. This is why the technique is called One-Hot Encoding.

As a result, a single categorical column may expand into multiple columns depending on the number of unique values it contains.

In the OneHotEncoding code above, the following code converts the encoded output into a DataFrame and assigns meaningful column names:

encodedDF = pd.DataFrame(encodedCols, 
                         columns=encoder.get_feature_names_out(categorical_cols), 
                         index=X_train.index)

The following code merges those newly created columns back into the training dataset:

# Replace the current values of the attribute with the encoded values
X_train = X_train.drop(columns=categorical_cols)
X_train = pd.concat([X_train, encodedDF], axis=1);

That’s enough about encoding for now!

Note: If you fit an encoder on the training dataset, you must use that same fitted encoder to transform your validation and test datasets. Never fit a new encoder on validation or test data, as this can lead to inconsistent feature representations and data leakage.

Using multiple features

To continue with the problem we are trying to solve, we will now extend our model by introducing multiple features. This allows the model to learn richer patterns from the data compared to using a single feature.

We define our target variable and feature set as follows:

target = dataFrame['exam_score']
features = ['study_hours', 'class_attendance', 'exam_difficulty']

all_train_data = dataFrame[features].copy()

Splitting the Dataset

Next, we split the dataset into training and validation sets. This helps us evaluate how well the model generalizes to unseen data.

from sklearn.model_selection import train_test_split

# Split data
train_data, val_data, train_target, val_target = train_test_split(
 all_train_data, 
 target, 
 test_size=0.2, 
 random_state=42
)

At this point, train_data contains both numerical and categorical features, which means we still need to preprocess the categorical columns before training the model.

Encoding Categorical Features

Since machine learning models require numerical input, we need to convert categorical features into numeric form. In this case, we use Ordinal Encoding for the exam_difficulty column.

from sklearn.preprocessing import OrdinalEncoder

# Encode categorical features
encoder = OrdinalEncoder()

# Manually select categorical attribute(s)
categorical_cols = ['exam_difficulty']

# Fit encoder on training data only
train_data[categorical_cols] = encoder.fit_transform(train_data[categorical_cols])

# Transform validation data using the same encoder
val_data[categorical_cols] = encoder.transform(val_data[categorical_cols])

Important Concept: Fit vs Transform

It is important to note the difference between fit_transform() and transform():

• fit_transform() is used on the training data because the encoder learns the mapping from categories to numbers.
• transform() is used on validation data to ensure it uses the same learned mapping.

Sample Data Flow

Before Encoding

At this point, the dataset still contains human-readable categorical values such as: easy, moderate, hard

These values are not yet usable by most machine learning algorithms. After encoding, they are converted into numerical representations that the model can process effectively

After encoding, it looks like this:

What’s Next?

The good news is that the code we previously wrote for creating and training our models (here) still works. The only change is that we now pass a dataset containing multiple features instead of a single feature.

As you experiment with the updated dataset, you may notice that the predictions appear to improve compared to the previous version. This is because the model now has access to more information that may help explain the relationship between the input features and the target variable.

However, visually inspecting predictions is not a reliable way to determine whether a model is performing well. As machine learning engineers, we need objective and measurable ways to evaluate our models.

This brings us to an important topic: Model Validation.

In the next article, we’ll explore how to properly evaluate machine learning models using validation metrics and techniques that help us measure performance, compare models, and make informed decisions about which model to use.

Until then, Keep debugging 💻!

GitHub Branch: The complete source code for this article is available here:

[embed]GitHub - wahabtaofeeqo/ml-tutorial at multiple-features Contribute to wahabtaofeeqo/ml-tutorial development by creating an account on GitHub.github.com