Feature Engineering & Model Evaluation — Day 4 Feature Selection Magic — Find Your Data’s Superstars!

Day 4 Feature Selection Magic — Find Your Data’s Superstars!

📖 Introduction: The Talent Show Analogy

Imagine: You’re a talent show judge with 100 contestants, but only 10 can perform. How do you choose? You’d pick the most talented ones, right?

Feature Selection is exactly this — choosing the most “talented” features (columns) from your dataset to help your model perform better!

Real-world example: When predicting house prices, do you need to know the house number? Or is square footage more important? Feature selection helps you decide!

🎯 Our Goal Today

By the end of this tutorial, you’ll be able to:

• Identify relevant features using correlation and mutual information
• Use Random Forest to find feature importance
• Choose the right feature selection method for your problem
• Build better models with fewer features

No prior knowledge needed — we’ll start from scratch! 🚀

📊 Let’s Create Our Practice Dataset

import pandas as pd
import numpy as np

# Create a sample housing dataset
np.random.seed(42)  # For reproducible results
data = {
    'house_size_sqft': np.random.randint(800, 3000, 100),
    'bedrooms': np.random.randint(1, 6, 100),
    'bathrooms': np.random.randint(1, 4, 100),
    'year_built': np.random.randint(1950, 2020, 100),
    'distance_to_city_km': np.random.uniform(1, 30, 100),
    'garden_size_sqft': np.random.randint(100, 1000, 100),
    'house_number': np.random.randint(1, 500, 100),  # Probably irrelevant!
    'random_noise': np.random.randn(100),  # Definitely irrelevant!
    'price': 0  # We'll calculate this based on meaningful features
}
# Create price based on meaningful features (plus some noise)
data['price'] = (data['house_size_sqft'] * 100 + 
                 data['bedrooms'] * 5000 + 
                 data['bathrooms'] * 3000 + 
                 np.random.normal(0, 10000, 100))
df = pd.DataFrame(data)
print("🏠 Our Housing Dataset Preview:")
print(df.head())
print(f"\nDataset shape: {df.shape}")

Output:

🏠 Our Housing Dataset Preview:
   house_size_sqft  bedrooms  bathrooms  year_built  distance_to_city_km  garden_size_sqft  house_number  random_noise      price
0             1492         4          2        1966            15.231234               456           423     -0.234153  244200.0
1             1868         3          1        1955             8.912345               789           245      0.542345  209800.0
2             1985         2          3        2005            22.123456               234           367     -1.123456  238500.0

Dataset shape: (100, 9)

We have 100 houses, 8 features, and 1 target variable (price). Some features are useful, some are probably useless!

📈 Section 1: Filter Methods — The Quick Auditions

What are Filter Methods?

Think of it like: Preliminary auditions where you quickly check each contestant’s basic skills without full performances.

1.1 Correlation Analysis

import seaborn as sns
import matplotlib.pyplot as plt

# Calculate correlation with target
correlation_with_price = df.corr()['price'].sort_values(ascending=False)
print("🎯 Correlation with Price:")
print(correlation_with_price)
# Visualize correlation matrix
plt.figure(figsize=(10, 8))
sns.heatmap(df.corr(), annot=True, cmap='coolwarm', center=0)
plt.title("Feature Correlation Matrix")
plt.show()

What this shows: How strongly each feature relates to price. Values close to 1 or -1 mean a strong relationship!

1.2 Mutual Information

from sklearn.feature_selection import mutual_info_regression

# Calculate mutual information
X = df.drop('price', axis=1)
y = df['price']
mi_scores = mutual_info_regression(X, y)
mi_series = pd.Series(mi_scores, index=X.columns).sort_values(ascending=False)
print("🔍 Mutual Information Scores:")
print(mi_series)

Simple explanation: Mutual information measures how much knowing a feature reduces uncertainty about the price. Higher = more informative!

🌳 Section 2: Embedded Methods — The Live Performances

What are Embedded Methods?

Think of it like: The actual talent show where contestants perform, and judges score them based on actual performance.

2.1 Random Forest Feature Importance

from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split

# Split data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Train Random Forest
rf = RandomForestRegressor(n_estimators=100, random_state=42)
rf.fit(X_train, y_train)
# Get feature importance
feature_importance = pd.Series(rf.feature_importances_, index=X.columns).sort_values(ascending=False)
print("🌳 Random Forest Feature Importance:")
print(feature_importance)
# Visualize
plt.figure(figsize=(10, 6))
feature_importance.plot(kind='bar')
plt.title("Feature Importance from Random Forest")
plt.ylabel("Importance Score")
plt.show()

What this does: The Random Forest actually “uses” each feature to make predictions and tells us which were most helpful!

📊 Section 3: Comparing All Methods

# Create comparison DataFrame
selection_results = pd.DataFrame({
    'Correlation': df.corr()['price'].drop('price'),
    'Mutual_Info': mi_series,
    'RF_Importance': feature_importance
})

# Normalize scores for comparison
selection_results = selection_results.apply(lambda x: x/x.max(), axis=0)
print("📊 Feature Selection Methods Comparison:")
print(selection_results.sort_values('RF_Importance', ascending=False))
# Plot comparison
selection_results.sort_values('RF_Importance', ascending=True).plot(kind='barh', figsize=(12, 8))
plt.title("Comparison of Feature Selection Methods (Normalized)")
plt.xlabel("Normalized Score")
plt.show()

This comparison shows: Which features consistently rank high across different methods — these are your true superstars! 🏆

🎯 Section 4: Decision Guide — Which Method to Use?

Simple Rules for Choosing:

def feature_selection_guide(scenario):
    """
    Simple decision guide for feature selection methods
    """
    rules = {
        'quick_analysis': "Use Correlation + Mutual Information (Filter Methods)",
        'accurate_importance': "Use Random Forest (Embedded Method)", 
        'many_features': "Start with Filter Methods, then use Embedded",
        'small_dataset': "Use Embedded Methods (they use the target variable)",
        'before_complex_model': "Use Filter Methods for initial feature reduction"
    }
    return rules.get(scenario, "Use Random Forest - it's usually the best!")

# Examples
print("For quick analysis:", feature_selection_guide('quick_analysis'))
print("For accurate results:", feature_selection_guide('accurate_importance'))
print("For many features:", feature_selection_guide('many_features'))

When to Use Each Method:

🚨 Section 5: Common Mistakes to Avoid

Mistake 1: Using Correlation Alone

# ❌ WRONG: Only using correlation
important_features = df.corr()['price'].abs().sort_values(ascending=False).head(3).index.tolist()

# ✅ CORRECT: Using multiple methods
correlation_features = df.corr()['price'].abs().sort_values(ascending=False).head(5).index.tolist()
mi_features = mi_series.head(5).index.tolist()
rf_features = feature_importance.head(5).index.tolist()
# Combine insights
final_features = list(set(correlation_features + mi_features + rf_features))

Mistake 2: Feature Selection Before Train-Test Split

# ❌ WRONG: Doing feature selection on entire dataset
all_data_importance = calculate_importance(df)  # Data leakage!

# ✅ CORRECT: Feature selection only on training data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
train_importance = calculate_importance(pd.concat([X_train, y_train], axis=1))

Mistake 3: Ignoring Domain Knowledge

# ❌ WRONG: Blindly trusting feature importance
# Might select 'house_number' if it accidentally correlates with price

# ✅ CORRECT: Combine statistical methods with domain knowledge
# "Does it make sense that house number affects price? Probably not!"

📈 Section 6: Putting It All Together — Final Feature Set

Let's select our top features based on all methods

top_features = feature_importance.head(4).index.tolist() # Top 4 from Random Forest

print("🎯 Selected Top Features:")
print(top_features)

# Create new dataset with only important features
df_optimized = df[top_features + ['price']]

print("\n📊 Optimized Dataset Shape:", df_optimized.shape)
print("🔥 Reduced from 8 to", len(top_features), "features!")
print("\nOptimized Dataset Preview:")
print(df_optimized.head())

Results: You’ll typically see 50–80% feature reduction while maintaining (or even improving) model performance! 🚀

🌍 Section 7: Real-World Applications

Where Feature Selection is Used:

1. Healthcare: Selecting vital signs that actually predict diseases
2. Finance: Choosing economic indicators that predict stock prices
3. E-commerce: Picking customer behaviors that predict purchases
4. Manufacturing: Identifying process parameters that affect quality

Success Story: A bank reduced 200 customer features to 15, making their loan approval model both more accurate and easier to explain to customers!

🏋️ Section 8: Practice Exercise

Your Turn to Practice!

# TODO: Create your own practice dataset
practice_data = {
    'feature_A': np.random.rand(50),
    'feature_B': np.random.rand(50), 
    'feature_C': np.random.rand(50),
    'feature_D': np.random.rand(50),
    'target': 0  # You'll create this!
}

# Create target that depends on feature_A and feature_C
practice_df = pd.DataFrame(practice_data)
practice_df['target'] = practice_df['feature_A'] * 100 + practice_df['feature_C'] * 50 + np.random.normal(0, 10, 50)

# TODO: Apply what you learned!
# 1. Calculate correlation with target
# 2. Compute mutual information scores  
# 3. Train Random Forest and get feature importance
# 4. Identify which features are actually important!

# HINT: The important features should be feature_A and feature_C!

📋 Cheat Sheet

Feature Selection Methods:

1. Filter Methods: Quick stats-based selection

• Correlation: df.corr()['target']
• Mutual Info: mutual_info_regression(X, y)

2. Embedded Methods: Model-based selection

• Random Forest: model.feature_importances_

When to Use:

• Quick analysis → Correlation + Mutual Info
• Accurate results → Random Forest
• Many features → Filter first, then Embedded

Golden Rules:

• Always validate with multiple methods
• Never select features using test data
• Combine stats with Doma

🎉 Conclusion

What you accomplished today:

• ✅ Learned 3 powerful feature selection methods
• ✅ Discovered how to find your data’s “superstars”
• ✅ Avoided common beginner mistakes
• ✅ Built a streamlined dataset for better models

Key takeaway: You don’t need all features — just the right ones! Feature selection is like giving your model a magnifying glass to focus on what truly matters. 🔍

Remember: In the world of data, quality beats quantity every time! Happy feature hunting! 🎯