Feature Selection Techniques in Machine Learning

Introduction

In Machine Learning, having more features does not always improve model performance.

Many datasets contain:

• irrelevant features
• duplicate information
• noisy columns

These unnecessary features can reduce model accuracy and increase training time.

That’s where Feature Selection becomes important.

Feature Selection helps us choose only the most useful features for training a Machine Learning model.

In this article, we will learn:

✅ What Feature Selection is ✅ Why Feature Selection matters ✅ Filter, Wrapper, and Embedded Methods ✅ Real-world Python examples ✅ Common beginner mistakes

What is Feature Selection?

Feature Selection is the process of selecting important input variables while removing irrelevant features.

Example:

Suppose we are predicting house prices.

Useful features:

• Area
• Location
• Number of Bedrooms

Unnecessary features:

• House ID
• Owner Phone Number

Removing unnecessary features helps the model focus on meaningful patterns.

Why Feature Selection is Important

FIG.2

Without feature selection:

• models become slower
• overfitting increases
• accuracy may decrease
• computation cost increases

Benefits of Feature Selection:

✅ Faster Training ✅ Better Accuracy ✅ Reduced Overfitting ✅ Simpler Models ✅ Better Interpretability

Types of Feature Selection Techniques

Feature Selection methods are mainly divided into:

1. Filter Methods
2. Wrapper Methods
3. Embedded Methods

A. Correlation Method

Correlation measures the relationship between variables.

Highly correlated features are more useful for prediction.

Correlation Heatmap

FIG.3

import seaborn as sns
import matplotlib.pyplot as plt

corr = data.corr()

sns.heatmap(corr, annot=True)
plt.show()

Advantages

• Very fast
• Easy to understand

Disadvantages

• Captures only linear relationships

B. Chi-Square Test

Chi-Square is used for categorical features.

It checks whether a feature is related to the target variable.

Visualization

FIG.4

from sklearn.feature_selection import SelectKBest
from sklearn.feature_selection import chi2

X_new = SelectKBest(score_func=chi2, k=5)

X_selected = X_new.fit_transform(X, y)

2. Wrapper Methods

Wrapper methods test multiple feature combinations and select the best subset.

These methods usually provide better accuracy but are computationally expensive.

A. Forward Selection

Forward Selection starts with zero features and adds important features one by one.

Process

1. Start with no features
2. Add the best feature
3. Repeat until performance stops improving

FIG.5

B. Backward Elimination

Backward Elimination starts with all features and removes the least important features step by step.

Python Example

import statsmodels.api as sm

X = sm.add_constant(X)

model = sm.OLS(y, X).fit()

print(model.summary())

C. Recursive Feature Elimination (RFE)

RFE repeatedly removes weak features until the desired number remains.

RFE Visualization

FIG.6

from sklearn.feature_selection import RFE
from sklearn.linear_model import LogisticRegression

model = LogisticRegression()

rfe = RFE(model, n_features_to_select=5)

X_rfe = rfe.fit_transform(X, y)

3. Embedded Methods

Embedded methods perform feature selection during model training.

These methods combine:

• model training
• feature selection

at the same time.

A. Lasso Regression

Lasso Regression removes unnecessary features by shrinking coefficients toward zero.

Loss=RSS+λ∑∣βj∣\text{Loss}=RSS+\lambda\sum |\beta_j|Loss=RSS+λ∑∣βj​∣

Lasso Visualization

FIG.7

from sklearn.linear_model import Lasso

model = Lasso(alpha=0.1)

model.fit(X, y)

B. Tree-Based Feature Importance

Decision Trees and Random Forest automatically calculate feature importance.

Visualization

FIG.8

from sklearn.ensemble import RandomForestClassifier

model = RandomForestClassifier()

model.fit(X, y)

print(model.feature_importances_)

Common Beginner Mistakes

1. Removing Features Randomly

Never remove columns without analysis.

Always check:

• correlation
• importance score
• domain knowledge

2. Using Too Many Features

More features can increase overfitting.

Sometimes fewer features produce better predictions.

3. Ignoring Data Leakage

Using future information during training creates unrealistic accuracy.

Always avoid feature leakage.

Real-World Applications

Feature Selection is widely used in:

• Fraud Detection
• Healthcare
• Recommendation Systems
• Stock Market Prediction
• NLP
• Image Recognition

Example: Medical datasets may contain thousands of features, but only a few are useful for diagnosis.

Conclusion

Feature Selection is one of the most important steps in Machine Learning.

It helps:

• improve model accuracy
• reduce overfitting
• speed up training
• simplify models

In this article, we explored:

• Filter Methods
• Wrapper Methods
• Embedded Methods
• Correlation
• RFE
• Lasso Regression
• Feature Importance

A good Machine Learning model is not built using more features — it is built using the right features.