Photo by Luke Jones on Unsplash

Data Preprocessing in Python: From Raw CSV to Machine Learning Ready Dataset

A hands-on guide to cleaning, transforming, and preparing data for AI using Python and Pandas

Learn how to preprocess data for machine learning using Python and Pandas. This step-by-step tutorial covers handling missing values, removing duplicates, feature selection, and encoding categorical variables.

Photo by fabio on Unsplash

Data preprocessing is the foundation of every successful machine learning project. No matter how powerful your model is, it will underperform on raw, messy data. In this hands-on tutorial, I’ll walk you through a complete data preprocessing pipeline in Python using a real-world style dataset — HousePricePrediction.csv — inside Google Colab.

By the end of this guide, you’ll know how to take a raw CSV file and transform it into a clean, structured, machine-learning-ready dataset.

Why Data Preprocessing Is Critical for Machine Learning

Real-world datasets almost always come with problems:

• Missing values that break model training
• Duplicate records that introduce bias
• Inconsistent formatting across columns
• Categorical text values that ML models can’t interpret
• Irrelevant features that add noise without adding signal

Data preprocessing solves all of these problems before you ever touch a model.

Tools and Libraries Used

• Python
• Pandas
• NumPy
• Google Colab

Step 1: Import Required Libraries

import pandas as pd
import numpy as np

Pandas is the go-to library for working with tabular data in Python. NumPy supports efficient numerical computations. These two libraries form the backbone of most data science workflows.

Step 2: Load the Dataset into a Pandas DataFrame

df = pd.read_csv('/content/HousePricePrediction.csv')
print(df.head(10))

Loading the dataset with read_csv() creates a DataFrame — a table-like structure that makes data manipulation easy. Printing the first 10 rows gives you an instant preview of your data's structure, column names, and value types.

Step 3: Explore and Understand the Dataset

Before cleaning anything, you need to understand what you’re working with.

Check the Shape (Rows × Columns)

print(df.shape)

Check Data Types

print(df.dtypes)

This distinguishes between numerical columns (int64, float64) and categorical/text columns (object). Knowing your data types early prevents many downstream errors.

Generate Summary Statistics

print(df.describe())

describe() returns count, mean, standard deviation, min, max, and quartile values — a quick statistical snapshot of your dataset.

Step 4: Identify Missing Values

Missing values are one of the most common data quality issues in real datasets.

print(df.isnull().sum())

This prints the count of missing values per column, so you know exactly where gaps exist before deciding how to fill them.

Step 5: Handle Missing Values

Fill Numerical Columns with the Median

numerical_cols = df.select_dtypes(include=['int64', 'float64']).columns

for col in numerical_cols:
    df[col] = df[col].fillna(df[col].median())

Why the median instead of the mean? The median is resistant to outliers. In housing datasets, extreme property prices can skew the mean significantly, making the median a safer and more representative imputation strategy.

Fill Categorical Columns with the Mode

categorical_cols = df.select_dtypes(include=['object']).columns

for col in categorical_cols:
    df[col] = df[col].fillna(df[col].mode()[0])

For text/categorical columns, the mode (most frequent value) is the standard imputation method. It preserves the existing distribution of categories.

Step 6: Remove Duplicate Records

Duplicate rows create bias by overrepresenting certain data points during training.

Check for Duplicates

print(df.duplicated().sum())

Drop Duplicates

df = df.drop_duplicates()

This ensures every observation in your training set is unique, leading to more generalizable models.

Step 7: Feature Selection — Drop Irrelevant Columns

Not every column helps a machine learning model learn. Identifier columns like Id carry no predictive signal.

if 'Id' in df.columns:
    df = df.drop('Id', axis=1)

Removing noise features reduces model complexity and can actually improve performance.

Step 8: Encode Categorical Variables with One-Hot Encoding

Machine learning algorithms require numerical inputs. Text categories must be converted to numbers.

df_encoded = pd.get_dummies(df, drop_first=True)

get_dummies() applies One-Hot Encoding — it creates a binary column for each category. The drop_first=True parameter removes one redundant column per feature to avoid the dummy variable trap.

Complete Data Preprocessing Pipeline (Full Code)

import pandas as pd
import numpy as np

# Load dataset
df = pd.read_csv('/content/HousePricePrediction.csv')

# Explore
print(df.head(10))
print(df.shape)
print(df.dtypes)
print(df.describe())

# Identify missing values
print(df.isnull().sum())

# Handle numerical missing values with median
numerical_cols = df.select_dtypes(include=['int64', 'float64']).columns
for col in numerical_cols:
    df[col] = df[col].fillna(df[col].median())

# Handle categorical missing values with mode
categorical_cols = df.select_dtypes(include=['object']).columns
for col in categorical_cols:
    df[col] = df[col].fillna(df[col].mode()[0])

# Remove duplicate records
df = df.drop_duplicates()

# Drop the Id column (non-predictive)
if 'Id' in df.columns:
    df = df.drop('Id', axis=1)

# One-hot encode categorical variables
df_encoded = pd.get_dummies(df, drop_first=True)

# Preview cleaned dataset
print(df_encoded.head())

The Preprocessed Dataset: What Changed?

After running this pipeline, the dataset is:

• ✅ Free of missing values
• ✅ Free of duplicate rows
• ✅ Stripped of non-predictive identifier columns
• ✅ Fully numerical (all categories encoded)
• ✅ Ready to be split into training and test sets

Key Takeaways

Preprocessing Step Technique Used Why It Matters Missing numerical values Median imputation Robust to outliers Missing categorical values Mode imputation Preserves category distribution Duplicate rows drop_duplicates() Removes training bias Irrelevant features Column drop Reduces noise Categorical encoding One-hot encoding Makes data ML-compatible

What’s Next After Preprocessing?

With a clean dataset, you’re ready to move into model building. Typical next steps include:

1. Feature scaling — Normalize or standardize numerical features using StandardScaler or MinMaxScaler
2. Train-test split — Use sklearn.model_selection.train_test_split()
3. Model training — Try a baseline model like Linear Regression or Random Forest
4. Evaluation — Measure performance with RMSE, MAE, or R² score

Frequently Asked Questions

Q: Why use median instead of mean for missing values? A: The mean is sensitive to outliers. A single extreme house price can pull the mean far from what’s typical, leading to poor imputations. The median stays stable regardless of outliers.

Q: What is One-Hot Encoding and when should I use it? A: One-Hot Encoding converts categorical text columns into binary (0/1) columns. Use it for nominal categories with no inherent order (like neighborhood names or house styles). For ordinal categories with a natural order, consider label encoding instead.

Q: Should I always drop the Id column? A: Yes — any column that uniquely identifies a row (like an ID or timestamp) provides zero predictive value to a model and should be removed.

Final Thoughts

Data preprocessing isn’t glamorous, but it’s where good machine learning is won or lost. The techniques covered here — imputation, deduplication, feature selection, and encoding — are used in real data science workflows every day.

If you found this useful, follow along as I continue documenting my journey through Python, Data Science, and Machine Learning.