Your first guide to Exploratory Data Analysis (EDA) — Python code included!

In my last article Loading Data Into Python we fought the FileNotFoundError boss and we won! You now have a variable named df sitting in your RAM. But here is the hard truth: Raw data tell lies!

It may look clean in Excel, but hidden inside are missing values, impossible numbers (like for examplle an age of 125 — maybe it is an error) and duplicates that will ruin your analysis.

If you feed this garbage into a machine learning model, you will get garbage out (the “Garbage In, Garbage Out” law).

Image made by me.

Welcome to Exploratory Data Analysis (EDA). This isn’t just about making pretty charts. It’s about the absolute, non-negotiable essential part of data analysis.

We are going to shine a light on the dark raw dataset until it confesses its secrets and we’re going to it with the simplest, step-by-step way. So, grab a drink or a snack and stay with me.

Step 1: Importing the needed libraries

You need more than just Pandas now. You also need visualization libraries to see the problems.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

# Make sure charts appear in the notebook
%matplotlib inline

• Pandas/Numpy: The number crunchers.
• Matplotlib: The foundation of plotting.
• Seaborn: Matplotlib’s cooler, better-dressed cousin. It makes complex plots (like heatmaps) easy and prettier!

Step 2: Take a look and understand your dataset (what’s your dealing with?)

Most tutorials out there tell you to run df.head(). I also use to do it that way, but I have learned also something else. Don't trust head(). The top 5 or 10 rows are often sorted or perfectly filled out. They hide the chaos at row 50,000.

The Better Way: df.sample()

Ask for random rows. This gives you a true snapshot of a messy dataset.

# Show 10 random rows
df.sample(10)

The Dataset Scan: df.info()

This is the single most important command in EDA. It tells you the data types (Dtype) and how much data is missing (Non-Null Count).

df.info()

What to look for:

1. The Nulls: If you have 10,000 rows but the “Age” column only has 8,000 non-null values, you have a “Swiss Cheese” (a lot of holes) problem.
2. The Wrong Data Types: Does your “Price” column say object instead of float? That means someone put a dollar sign ($) or a comma in there, and Python treats it as text. You can't do math on text.

The Dataset Summary: df.describe()

This gives you the basic stats (mean, min, max, std) for numerical columns.

df.describe()

The Detective Check: Look at the min and max.

• Is the minimum Age -1? (Impossible).
• Is the max Income 999999999? (Likely a placeholder for "error").

Step 3: Cleanning the data

You found NaN (Not a Number) values in step 2. Now you have to deal with them.

A: Quantify the Damage

Don’t guess. Count them.

# How many missing values in each column?
print(df.isnull().sum())

# Want it as a percentage?
print(df.isnull().mean() * 100)

The percentage may help as to choose the right solution, as you can see right below.

B: The Fixes

You have three choices, ranked by severity:

1. The Nuclear Option (Drop): If a row is mostly empty, kill it.

df.dropna(inplace=True)

! Use when: You have huge data and few missing rows (usually less than 5%).

2. The “Average” Approach (Impute Mean/Median): Fill the holes with the average. This is often the most used option (especialy if the percentage of missing values don’t allow you to drom them).

# Use Median if you have outliers (billionaires skew the mean income)
df['Age'].fillna(df['Age'].median(), inplace=True)

3. The “Unknown” Label: For categorical data (text), just be honest.

df['Gender'].fillna('Unknown', inplace=True)

The “Twin” Problem: Duplicates

Duplicate rows are silent killers. They bias your model by telling it that a specific event happened twice when it only happened once

# 1. Check for dublicates
print({df.duplicated().sum()})

# 2. Destroy them
df.drop_duplicates(inplace=True)

Step 4: Visualizing the data (One image is worth a thousand words)

Numbers in a table are hard to read. Shapes are easy to understand.

(This part can be much more detailed with code containing many arguments and parameters, but I will keep it as simlpe as possible.)

The Histogram (For Numbers)

This shows you the shape of your data. Is it a Bell Curve (Normal Distribution)? Is it skewed?

sns.histplot(df['Age'], kde=True)
plt.show()

The Insight: If your chart leans heavily to the left, but has a long tail to the right, using the “Mean” for missing values is dangerous. Use the “Median.”

https://www.researchgate.net/figure/Histograms-and-estimated-pdfs-of-the-normal-and-skew-normal-fit-to-the-length-and_fig2_286882582

The Count Plot (For Categories)

How many men vs. women? How many sales per country?

sns.countplot(x='Country', data=df)
plt.xticks(rotation=45) # Rotate text so it doesn't overlap
plt.show()

The Insight: If 99% of your data is from “USA” and 1% is “France,” your model will be biased. You might need to fix this imbalance later.

The Correlation Heatmap (The Relationships: “Who is dating who?”)

This is the “Money Plot.” It shows which variables move together.

# Calculate correlation matrix
corr = df.corr()

# Plot it
sns.heatmap(corr, annot=True, cmap='coolwarm')

https://medium.com/@szabo.bibor/how-to-create-a-seaborn-correlation-heatmap-in-python-834c0686b88e

How to read it:

• 1.0: Perfect positive match (As Height goes up, Weight goes up).
• -1.0: Perfect negative match (As Car Weight goes up, Speed goes down).
• 0: No relationship.

The Boxplot (The Outlier Hunter)

This shows the spread of data and dots for outliers.

sns.boxplot(x='Gender', y='Salary', data=df)

• The Insight: Are the dots way above the box? Those are your outliers. You need to decide if you keep them or drop them.

https://www.kdnuggets.com/2019/11/understanding-boxplots.html

Step 5: Feature Engineering

This is where a Data Analyst becomes a Data Scientist. Feature Engineering is the art of creating new information from existing data!

Scenario A: The DateTime

Your column is “2023–01–01”. The computer sees a string. You want the “Month” or “Day of Week.”

# Convert to datetime first!
df['Date'] = pd.to_datetime(df['Date'])

# Extract features
df['Month'] = df['Date'].dt.month
df['DayOfWeek'] = df['Date'].dt.day_name()

Scenario B: Binning (The “Bucket” Strategy)

Detailed numbers can be noisy. sometimes grouping them helps.

# Turn exact ages (24, 25, 26) into groups (Young, Adult, Senior)
df['Age_Group'] = pd.cut(df['Age'], bins=[0, 18, 65, 100], labels=['Child', 'Adult', 'Senior'])

https://www.mygreatlearning.com/blog/what-is-feature-engineering/

Summary Checklist

Before you move to modeling, answer these 5 questions:

1. Shape: How many rows/columns do I have?
2. Types: Are my numbers actually numbers?
3. Missing: Did I fill or drop the NaNs?
4. Outliers: Did I spot the billionaire in the room?
5. Features: Did I extract the Month from the Date?

Once you check these boxes, you aren’t just guessing anymore. You are analyzing.

Next Up: We take this clean data and start predicting the future (Intro to Machine Learning Models).

I hope that this article was helpful. Read more from me here.

Happy Coding!

Meme made on imgflip.com