Powerful EDA Techniques That Separate Beginners from Experts

Most data scientists skip this step. The best ones never do. https://www.linkedin.com/in/shorya-bisht-a20144349/

You just downloaded a fresh dataset. What’s the first thing you do?

If you answered “build a model” — this post is for you.

There’s a quiet but gaping divide between data scientists who get results and those who get confused. It doesn’t live in the complexity of their models or the size of their GPUs. It lives in those first, unglamorous hours spent actually looking at the data — poking it, squinting at it, asking it uncomfortable questions.

That process has a name: Exploratory Data Analysis, or EDA.

“Exploratory data analysis can never be the whole story, but nothing else can serve as the foundation stone — as the first step.” — John W. Tukey, the statistician who coined the term EDA

Beginners treat EDA like a formality — a few .head() calls and a shape check before jumping to the "real work." Experts treat it like detective work. And the difference in outcomes is staggering.

In this post, we’re going to walk through the EDA techniques that actually matter — using one of the most familiar datasets in machine learning — and by the end, you’ll never look at raw data the same way again.

What Is EDA, Really? (And Why Should You Care?)

Imagine you’re a doctor and a patient walks in. Do you immediately prescribe medication? No — you ask questions, run tests, check vitals, look for patterns, and then make a diagnosis.

EDA is the doctor’s examination. Your dataset is the patient.

At its core, EDA is about:

1. Understanding what you have — data types, size, structure
2. Finding what’s broken — missing values, outliers, inconsistencies
3. Discovering what’s interesting — trends, patterns, relationships
4. Forming hypotheses — guesses worth testing with a model

“Without EDA, your model is a blind archer. With it, you at least know which direction to aim.” — Hadley Wickham, Chief Scientist at Posit (formerly RStudio)

Our Dataset: The Titanic

We’re using the Titanic dataset — a beloved classic in the ML community. It contains passenger information from the 1912 Titanic disaster and asks one haunting question: who survived, and why?

It has the perfect mix of numerical, categorical, and missing data — making it an ideal EDA playground.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

# Load the dataset
df = pd.read_csv('https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv')

print(df.shape)
df.head()

(891, 12)

We have 891 rows and 12 columns. Now the real work begins.

Step 1 — The First Look (What Do You Actually Have?)

Analogy: Opening a Mystery Box

Think of this as opening a mystery box you received in the mail. Before doing anything with the contents, you’d naturally take inventory — count the items, check for anything broken, look at what types of things are in there.

# Data types and non-null counts
df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 891 entries, 0 to 890
Data columns (total 12 columns):
 #   Column       Non-Null Count  Dtype  
---  ------       --------------  -----  
 0   PassengerId  891 non-null    int64  
 1   Survived     891 non-null    int64  
 2   Pclass       891 non-null    int64  
 3   Name         891 non-null    object 
 4   Sex          891 non-null    object 
 5   Age          714 non-null    float64   ← Missing 177 values!
 6   SibSp        891 non-null    int64  
 7   Parch        891 non-null    int64  
 8   Ticket       891 non-null    object 
 9   Fare         891 non-null    float64
10   Cabin        204 non-null    object    ← Missing 687 values!
11   Embarked     889 non-null    object    ← Missing 2 values

# Summary statistics
df.describe()

         Survived      Pclass         Age       SibSp       Parch        Fare
count    891.000000  891.000000  714.000000  891.000000  891.000000  891.000000
mean       0.383838    2.308642   29.699118    0.523008    0.381594   32.204208
std        0.486592    0.836071   14.526497    1.102743    0.806057   49.693429
min        0.000000    1.000000    0.420000    0.000000    0.000000    0.000000
25%        0.000000    2.000000   20.125000    0.000000    0.000000    7.910400
50%        0.000000    3.000000   28.000000    0.000000    0.000000   14.454200
75%        1.000000    3.000000   38.000000    1.000000    0.000000   31.000000
max        1.000000    3.000000   80.000000    8.000000    6.000000  512.329200

What a beginner sees: Numbers.

What an expert sees: Stories.

Notice: Only 38.4% of passengers survived. The average age was ~30. Fare ranges from 0 to $512 — that’s an enormous spread, which hints at major economic disparity. These aren’t just statistics. They’re context.

Step 2 — Missing Data Analysis (The Holes in Your Story)

Analogy: A Book with Torn Pages

Missing data is like reading a thriller novel with pages randomly torn out. You can still follow the story — but the gaps might be hiding crucial plot twists.

# Visualize missing data
missing = df.isnull().sum().sort_values(ascending=False)
missing_pct = (missing / len(df)) * 100

missing_df = pd.DataFrame({
    'Missing Count': missing,
    'Missing %': missing_pct
}).query('`Missing Count` > 0')

print(missing_df)

# Heatmap of missing values
plt.figure(figsize=(10, 6))
sns.heatmap(df.isnull(), cbar=False, cmap='viridis', yticklabels=False)
plt.title('Missing Value Map — Yellow = Missing', fontsize=14)
plt.tight_layout()
plt.show()

Missing Count  Missing %
Cabin               687      77.10   ← Nearly useless as-is
Age                 177      19.87   ← Imputable
Embarked              2       0.22   ← Easy to fix

Expert Insight: Not all missing data is equal.

• 77% missing (Cabin): Likely structurally missing — lower class passengers often had no cabin assignment. This missingness itself is a signal.
• 20% missing (Age): Random-ish. Can be imputed with median or model-based approaches.
• 0.22% missing (Embarked): So small, you can safely fill with the mode or drop those rows.

# Quick fix for small missing values
df['Embarked'].fillna(df['Embarked'].mode()[0], inplace=True)

# Create a binary flag before imputing Age — the missingness might matter!
df['Age_was_missing'] = df['Age'].isnull().astype(int)
df['Age'].fillna(df['Age'].median(), inplace=True)

# Turn Cabin into a binary "had cabin / no cabin"
df['Has_Cabin'] = df['Cabin'].notnull().astype(int)

That Age_was_missing flag is an expert move. You're not just filling the hole — you're remembering there was a hole. Sometimes who has missing data is as informative as the data itself.

Step 3 — Univariate Analysis (Getting to Know Each Variable)

Analogy: Learning About Each Team Member Before the Game

Before analyzing how a sports team performs together, you study each player’s stats individually. Same idea here.

fig, axes = plt.subplots(1, 2, figsize=(14, 5))

# Survival distribution
df['Survived'].value_counts().plot(kind='bar', ax=axes[0],
    color=['#e74c3c', '#2ecc71'], edgecolor='black')
axes[0].set_title('Survival Count', fontsize=13)
axes[0].set_xticklabels(['Did Not Survive', 'Survived'], rotation=0)
axes[0].set_ylabel('Count')

# Age distribution
axes[1].hist(df['Age'], bins=30, color='#3498db', edgecolor='white', alpha=0.8)
axes[1].set_title('Age Distribution', fontsize=13)
axes[1].set_xlabel('Age')
axes[1].set_ylabel('Frequency')

plt.tight_layout()
plt.show()

# Fare distribution — notice the heavy right skew
plt.figure(figsize=(10, 4))
plt.subplot(1, 2, 1)
plt.hist(df['Fare'], bins=50, color='#9b59b6', edgecolor='white')
plt.title('Fare Distribution (Raw)')

plt.subplot(1, 2, 2)
plt.hist(np.log1p(df['Fare']), bins=50, color='#e67e22', edgecolor='white')
plt.title('Fare Distribution (Log-Transformed)')
plt.tight_layout()
plt.show()

The raw Fare distribution is wildly right-skewed — a classic case where log transformation reveals a much cleaner, near-normal distribution. This matters when you feed it into algorithms that assume normality.

Interlude — Literature Review & Hypothesis Formation (The Step That Makes You a Scientist, Not Just a Coder)

Most tutorials skip this entirely. That’s exactly why most models underperform.

You’ve now seen your data’s shape. You know what’s missing. You under- stand how each individual variable behaves. At this point, a beginner opens a new notebook cell and starts plotting correlations. An expert does something different first — they think.

This interlude is about the intellectual work that happens between looking at data and drawing conclusions from it. It has two parts: reviewing what others already know, and forming your own educated guesses.

Part A — The Literature Review: Standing on Shoulders

“If I have seen further, it is by standing on the shoulders of giants.” — Isaac Newton

Newton was talking about science. But he could just as easily have been talking about data science.

A literature review in the EDA context means: before you hypothesize, find out what people who’ve studied this domain already know. It is the act of bring- ing external knowledge — research papers, domain expertise, prior analyses, historical context — into your investigation.

Analogy: The Detective Who Reads Case Files Imagine a detective assigned to a new case. A reckless detective jumps straight to interrogating suspects. A skilled one first reads the existing case files, checks prior incidents in the same neighborhood, and consults colleagues who’ve seen similar crimes. This prior knowledge doesn’t bias their investigation — it focuses it. They know which questions are worth asking.

That’s a literature review. Not to copy someone else’s conclusions, but to sharpen your own questions.

Why It Matters for the Titanic For the Titanic, a quick literature review would surface these known facts:

• Maritime law in 1912 operated under an informal “women and children first” protocol during evacuations

• The Titanic’s lifeboat capacity was only ~1,178 people for 2,224 aboard — a structural scarcity

• First-class passengers had cabins on upper decks, physically closer to the lifeboats

• Historical records confirm that crew members prioritized certain passen- gers during evacuation

These facts aren’t in the CSV. But they radically change what you look for. Without this context, a correlation between Sex and Survived is just a number. With it, it’s a confirmation of a documented historical bias.

Literature review informs WHICH features deserve deeper attention

Based on domain knowledge, we know these variables are historically significant:

priority_features = {

‘Sex’: ‘Women & children first protocol — expect strong gender effect’, ‘Pclass’: ‘Upper decks = closer to lifeboats, wealth = better access’,

‘Age’: ‘Children prioritized, elderly may have struggled with evacuation’, ‘Fare’: ‘Proxy for wealth and social class — correlates with deck location’, ‘Embarked’: ‘Different ports = different demographic compositions’

}

print(“Domain-Informed Feature Priority List”) print(“=” * 55)

for feature, rationale in priority_features.items():

print(f”\n [{feature}]\n → {rationale}”)

Domain-Informed Feature Priority List

=======================================================

[Sex]

→ Women & children first protocol — expect strong gender effect

[Pclass]

→ Upper decks = closer to lifeboats, wealth = better access

[Age]

→ Children prioritized, elderly may have struggled with evacuation

[Fare]

→ Proxy for wealth and social class — correlates with deck location

[Embarked]

→ Different ports = different demographic compositions

This is not a code output you’ll find in your DataFrame.

It’s a thinking output— and it’s just as important.

“A data scientist without domain knowledge is like a surgeon who’s only read anatomy books but never treated a patient. The data alone won’t tell you what to look for.” — DJ Patil, Former U.S. Chief Data Scientist

Part B — Hypothesis Formation: Educated Guessing as a Discipline

Now comes the moment where your observations and your literature review collide to produce something actionable: a hypothesis.

A hypothesis in EDA is a specific, testable statement about a relationship you expect to find in the data. It is not a vague feeling. It is not “I think age matters.” It is a precise claim that your subsequent analysis will either support, challenge, or complicate.

Analogy: The Weather Forecaster A weather forecaster doesn’t just stare at clouds and guess. They look at barometric pressure, temperature gradients, historical seasonal data, and satellite imagery — then make a specific, falsifiable prediction: “70% chance of rain between 3–6 PM.” If it doesn’t rain, they update their models. If it does, they gain confidence in their approach.

Your hypotheses work the same way. Make them specific. Then let the data verdict them.

The Anatomy of a Good Hypothesis A well-formed EDA hypothesis has three parts:

1. The claim — What you believe is true

2. The direction — Which way the relationship goes

3. The reasoning — Why you believe this (from domain knowledge or early observations)

Titanic Hypotheses — Formed from Observations + Literature

hypotheses = [

{

“id”: “H1”,

“claim”: “Female passengers had significantly higher survival rates than males”, “direction”: “Female Survived > Male Survived”,

“reasoning”: “Historical ‘women and children first’ maritime protocol”, “feature”: “Sex → Survived”,

“type”: “Categorical relationship”

},

{

“id”: “H2”,

“claim”: “First-class passengers survived at higher rates than third-class”, “direction”: “Pclass 1 Survived > Pclass 3 Survived”,

“reasoning”: “Upper deck cabins = physical proximity to lifeboats; wealth = influen “feature”: “Pclass → Survived”,

“type”: “Ordinal trend”

},

{

“id”: “H3”,

“claim”: “Children under 12 had higher survival rates than adults”, “direction”: “Age < 12 Survived > Age > 18 Survived”,

“reasoning”: “Children explicitly prioritized during evacuation”, “feature”: “Age → Survived”,

“type”: “Threshold effect”

},

{

“id”: “H4”,

“claim”: “Passengers travelling alone survived less than those with family”, “direction”: “Family_Size > 1 Survived > Family_Size == 1 Survived”,

“reasoning”: “Social support and coordination aid in evacuation; no one to advocate “feature”: “Family_Size → Survived”,

“type”: “Non-linear/U-shaped pattern”

},

{

“id”: “H5”,

“claim”: “The gender effect on survival will be stronger than the class effect”, “direction”: “Effect(Sex) > Effect(Pclass)”,

“reasoning”: “Protocol-driven evacuation overrides economic factors”, “feature”: “Sex vs Pclass — comparative”,

“type”: “Effect magnitude comparison”

}

]

# Print hypothesis register

print(f”{‘ID’:<5} {‘Claim’:<55} {‘Type’:❤0}”)

print(“-” * 95)

for h in hypotheses:

print(f”{h[‘id’]:<5} {h[‘claim’]:<55} {h[‘type’]:❤0}”)

ID Claim Type

H1 Female passengers had significantly higher survival… Categorical relationship

H2

First-class passengers survived at higher rates…

Ordinal trend

H3

Children under 12 had higher survival rates…

Threshold effect

H4

Passengers travelling alone survived less…

Non-linear/U-shaped pattern

H5

The gender effect will be stronger than class effect

Effect magnitude comparison

Now let’s test them — quickly, before we dive into full analysis:

# Quick hypothesis validation checks

print(“HYPOTHESIS VALIDATION SUMMARY”)

print(“=” * 55)

# H1 — Gender effect

gender_survival = df.groupby(‘Sex’)[‘Survived’].mean() print(f”\nH1 — Gender Effect:”) print(gender_survival.to_string())

verdict_h1 = “ SUPPORTED” if gender_survival[‘female’] > gender_survival[‘male’] else “ RE

print(f”Verdict: {verdict_h1}”)

# H2 — Class effect

class_survival = df.groupby(‘Pclass’)[‘Survived’].mean().sort_index() print(f”\nH2 — Class Effect:”)

print(class_survival.to_string())

verdict_h2 = “ SUPPORTED” if class_survival[1] > class_survival[3] else “ REJECTED” print(f”Verdict: {verdict_h2}”)

# H3 — Children effect

child_survival = df[df[‘Age’] < 12][‘Survived’].mean() adult_survival = df[df[‘Age’] >= 18][‘Survived’].mean() print(f”\nH3 — Children vs Adults:”)

print(f” Children (<12): {child_survival:.3f}”) print(f” Adults (>=18): {adult_survival:.3f}”)

verdict_h3 = “ SUPPORTED” if child_survival > adult_survival else “ REJECTED”

print(f”Verdict: {verdict_h3}”)

# H4 — Alone vs family

df[‘Family_Size’] = df[‘SibSp’] + df[‘Parch’] + 1 alone_survival = df[df[‘Family_Size’] == 1][‘Survived’].mean()

family_survival_rate = df[df[‘Family_Size’] > 1][‘Survived’].mean() print(f”\nH4 — Alone vs Family:”)

print(f” Alone: {alone_survival:.3f}”) print(f” With family:{family_survival_rate:.3f}”)

verdict_h4 = “ SUPPORTED” if family_survival_rate > alone_survival else “ REJECTED”

print(f”Verdict: {verdict_h4}”)

# H5 — Gender vs Class effect magnitude

from scipy.stats import pointbiserialr gender_binary = (df[‘Sex’] == ‘female’).astype(int)

corrgender, = pointbiserialr(gender_binary, df[‘Survived’])

corrclass, = pointbiserialr(-df[‘Pclass’], df[‘Survived’]) # Negative: higher class = l

print(f”\nH5 — Effect Magnitude (Point-Biserial Correlation):”) print(f” |Gender-Survival correlation|: {abs(corr_gender):.3f}”) print(f” |Class-Survival correlation|: {abs(corr_class):.3f}”)

verdict_h5 = “ SUPPORTED” if abs(corr_gender) > abs(corr_class) else “ REJECTED”

print(f”Verdict: {verdict_h5}”)

HYPOTHESIS VALIDATION SUMMARY

=======================================================

H1 — Gender Effect:

Sex

female 0.742038

male 0.188908 Verdict: SUPPORTED

H2 — Class Effect:

Pclass

1 0.629630

2 0.472826

3 0.242363

Verdict: SUPPORTED

H3 — Children vs Adults: Children (<12): 0.590909

Adults (>=18): 0.382398

Verdict: SUPPORTED

H4 — Alone vs Family: Alone: 0.303538

With family: 0.505650 Verdict: SUPPORTED

H5 — Effect Magnitude (Point-Biserial Correlation):

|Gender-Survival correlation|: 0.543

|Class-Survival correlation|: 0.338 Verdict: SUPPORTED

Five hypotheses. Five data-backed verdicts. All supported — but crucially, now you have magnitudes. Gender (0.543) has a 60% stronger effect on survival than class (0.338). That’s not just interesting — it tells you exactly where to focus your feature importance and model interpretation.

This is the difference between exploring data and interrogating it.

The Hypothesis → EDA → Insight Loop

The relationship between literature review, hypothesis formation, and EDA isn’t linear — it’s a loop:

Prior Knowledge (Literature Review)

↓

Form Hypotheses

↓

Run EDA Analysis ←

↓

Confirm / Reject / Refine

↓

New Questions Emerge

Every time you reject or partially support a hypothesis, you learn something

that generates a new, sharper one. That iterative loop is what separates an EDA that takes two hours from one that takes two days — and produces ten times the insight.

Step 4 — The Heart of It: Trends vs. Patterns

This is where most tutorials go quiet. But it’s arguably the most important conceptual distinction in EDA. Let’s break it down clearly.

What is a TREND?

A trend is a consistent directional movement in data over time or along an ordered axis.

Real-life analogy: Imagine watching your city’s temperature recordings over 12 months. You notice it gradually warms from January to July, then gradually cools back down. That gradual, directional change is a trend. It tells you: “things are moving this way.”

Trend = Direction over time or order.

In the Titanic dataset, while we don’t have a time axis, we can observe trends along ordered variables like Fare or Age:

# Survival rate across age bins — is there a trend?
df['Age_Bin'] = pd.cut(df['Age'], bins=[0, 12, 18, 35, 60, 80],
                        labels=['Child', 'Teen', 'Young Adult', 'Adult', 'Senior'])

age_survival = df.groupby('Age_Bin', observed=True)['Survived'].mean().reset_index()

plt.figure(figsize=(9, 5))
plt.plot(age_survival['Age_Bin'], age_survival['Survived'],
         marker='o', color='#e74c3c', linewidth=2.5, markersize=8)
plt.title('Survival Rate Across Age Groups — Spotting the Trend', fontsize=13)
plt.ylabel('Survival Rate')
plt.xlabel('Age Group')
plt.ylim(0, 1)
plt.grid(axis='y', linestyle='--', alpha=0.5)
plt.tight_layout()
plt.show()

What you’ll see: Children have the highest survival rate (~58%), which drops for teens, rises slightly for young adults, then falls for seniors. The general downward trend with age (after childhood) tells a story about evacuation priorities.

🔷 What is a PATTERN?

A pattern is a repeating structure, relationship, or arrangement in data — not necessarily directional, but consistently present.

Real-life analogy: Notice how traffic is always heavier on Monday mornings and Friday evenings — every week, without fail. That’s a pattern. It’s not going in one “direction” — it repeats. It tells you: “something is happening here, regularly.”

Pattern = Recurring structure or relationship.

# Survival pattern across Pclass and Sex
pivot = df.pivot_table(values='Survived', index='Pclass',
                        columns='Sex', aggfunc='mean')

plt.figure(figsize=(8, 5))
pivot.plot(kind='bar', color=['#e91e8c', '#1e90ff'],
           edgecolor='black', width=0.6)
plt.title('Survival Rate by Class & Gender — A Clear Pattern', fontsize=13)
plt.ylabel('Survival Rate')
plt.xlabel('Passenger Class')
plt.xticks(rotation=0, labels=['1st Class', '2nd Class', '3rd Class'])
plt.legend(title='Gender')
plt.grid(axis='y', linestyle='--', alpha=0.4)
plt.tight_layout()
plt.show()

What you’ll see: Regardless of class, females consistently outsurvived males. And within each gender, higher class = higher survival. This isn’t a direction — it’s a repeating relational structure. That’s a pattern.

The Key Difference — A Visual Summary

Trend Pattern Definition Directional movement over order/time Repeating structure or relationship Question it answers “Is this going up or down?” “Does this happen consistently?” Titanic Example Survival drops with age (after childhood) Women and 1st class always survive more Tool to extract Line plots, rolling averages Heatmaps, grouped bar charts, pivot tables What it implies Change is happening A rule or structure exists

Step 5 — Bivariate Analysis (How Variables Talk to Each Other)

# Correlation heatmap
plt.figure(figsize=(10, 7))
numeric_df = df.select_dtypes(include=[np.number])
corr = numeric_df.corr()

mask = np.triu(np.ones_like(corr, dtype=bool))
sns.heatmap(corr, mask=mask, annot=True, fmt='.2f',
            cmap='coolwarm', center=0, linewidths=0.5,
            cbar_kws={'shrink': 0.8})
plt.title('Correlation Matrix — Who Talks to Whom?', fontsize=14)
plt.tight_layout()
plt.show()

Expert Reading: Notice that Pclass has a -0.34 correlation with Survived (higher class number = lower survival) and Fare has +0.26 (higher fare = better survival). These two tell the same economic story from different angles. A beginner might treat them as independent signals; an expert recognizes them as correlated and accounts for multicollinearity.

# Box plots — great for spotting distribution differences between groups
fig, axes = plt.subplots(1, 2, figsize=(14, 6))

sns.boxplot(x='Survived', y='Age', data=df, palette='Set2', ax=axes[0])
axes[0].set_title('Age vs Survival')
axes[0].set_xticklabels(['Did Not Survive', 'Survived'])

sns.boxplot(x='Survived', y='Fare', data=df, palette='Set3', ax=axes[1])
axes[1].set_title('Fare vs Survival')
axes[1].set_xticklabels(['Did Not Survive', 'Survived'])

plt.tight_layout()
plt.show()

Survivors paid noticeably higher fares. The median fare for survivors is roughly double that of non-survivors. Economics, once again, telling the story clearly.

Step 6 — Multivariate Analysis (The Expert’s Territory)

Analogy: Reading a Room vs. a Single Person

A beginner reads one variable at a time (one person). An expert reads the whole room at once — catching how people interact, who influences whom, and which combinations matter.

# FacetGrid — survival by age, split by gender and class
g = sns.FacetGrid(df, col='Pclass', row='Sex',
                  hue='Survived', palette={0: '#e74c3c', 1: '#2ecc71'},
                  height=3.5, aspect=1.2)
g.map(plt.hist, 'Age', bins=20, alpha=0.7, edgecolor='white')
g.add_legend(title='Survived')
g.set_titles(row_template='{row_name}', col_template='Class {col_name}')
g.fig.suptitle('Age Distribution by Gender, Class & Survival', y=1.03, fontsize=14)
plt.tight_layout()
plt.show()

This single visualization tells you more than ten separate plots. You can simultaneously see: age distribution, gender differences, class differences, and survival outcomes — all in one grid.

# Pair plot for a holistic numerical view
cols = ['Survived', 'Age', 'Fare', 'Pclass', 'SibSp', 'Parch']
sns.pairplot(df[cols], hue='Survived',
             palette={0: '#e74c3c', 1: '#2ecc71'},
             plot_kws={'alpha': 0.5}, diag_kind='kde')
plt.suptitle('Pair Plot — Every Variable vs. Every Variable', y=1.01, fontsize=14)
plt.show()

Step 7 — Outlier Detection (The Troublemakers in Your Data)

Analogy: Finding the Outliers in a Classroom

Imagine a class of 30 students where 28 score between 50–80, one scores 12, and another scores 99. Those two students are outliers. They’re not necessarily wrong — they might be genuinely exceptional — but you need to know they exist before drawing conclusions about the class average.

# IQR-based outlier detection for Fare
Q1 = df['Fare'].quantile(0.25)
Q3 = df['Fare'].quantile(0.75)
IQR = Q3 - Q1

lower = Q1 - 1.5 * IQR
upper = Q3 + 1.5 * IQR

fare_outliers = df[(df['Fare'] < lower) | (df['Fare'] > upper)]
print(f"Number of Fare outliers: {len(fare_outliers)}")
print(f"Max fare: ${df['Fare'].max():.2f}")
print(f"Upper fence: ${upper:.2f}")

Number of Fare outliers: 116
Max fare: $512.33
Upper fence: $65.63

# Z-score method
from scipy import stats

df['Fare_zscore'] = np.abs(stats.zscore(df['Fare']))
extreme_outliers = df[df['Fare_zscore'] > 3]
print(f"\nExtreme outliers (|z| > 3): {len(extreme_outliers)} passengers")
print(extreme_outliers[['Name', 'Pclass', 'Fare']].head())

Expert decision: Don’t blindly remove outliers. That $512 fare likely belongs to a real first-class passenger. Removing it would erase genuine information. Instead, consider log-transforming the feature to reduce its influence.

Step 8 — Feature Engineering During EDA (Turning Observations into Gold)

“Feature engineering is the process of using domain knowledge to extract features from raw data. It is the art of transforming raw inputs into representations that algorithms can exploit.” — Pedro Domingos, Professor, University of Washington

EDA isn’t just about understanding data — it’s about creating new information from what you’ve observed.

# Extract title from Name — a hidden social status signal
df['Title'] = df['Name'].str.extract(r' ([A-Za-z]+)\.', expand=False)

print(df['Title'].value_counts().head(10))

Mr        517
Miss      182
Mrs       125
Master     40
Dr          7
Rev         6
...

# Group rare titles
title_map = {
    'Mr': 'Mr', 'Miss': 'Miss', 'Mrs': 'Mrs', 'Master': 'Master'
}
df['Title_Group'] = df['Title'].map(title_map).fillna('Other')

# Survival by title — immediate payoff
title_survival = df.groupby('Title_Group')['Survived'].mean().sort_values(ascending=False)
print(title_survival)

Title_Group
Miss      0.697802
Mrs       0.792000
Master    0.575000
Other     0.444444
Mr        0.156673

With one feature engineering step, you’ve created a variable where “Mr” has only a 15.7% survival rate while “Mrs” has 79.2%. That’s extraordinarily predictive — and it was hiding in the Name column the whole time.

# Family size — another derived feature
df['Family_Size'] = df['SibSp'] + df['Parch'] + 1  # +1 for self
df['Is_Alone'] = (df['Family_Size'] == 1).astype(int)

# Survival by family size
family_survival = df.groupby('Family_Size')['Survived'].mean()
print(family_survival)

Family_Size
1     0.303538   ← Alone — low survival
2     0.552795   ← Small family — best survival
3     0.578431
4     0.724138
5     0.200000   ← Large family — harder to evacuate
6     0.136364
7     0.333333
8     0.000000
11    0.000000

Medium-sized families had the best survival odds. Being alone was bad; being in a massive family was catastrophic. This U-shaped pattern would have been invisible without EDA-guided feature engineering.

Step 9 — The EDA Report: Summarizing Your Findings

A professional EDA always ends with a structured summary. Here’s how experts document findings:

print("=" * 60)
print("EDA SUMMARY REPORT — TITANIC DATASET")
print("=" * 60)

print(f"\n📦 Dataset: {df.shape[0]} rows × {df.shape[1]} columns")
print(f"🎯 Target Variable: Survived (Binary: 0/1)")
print(f"📊 Survival Rate: {df['Survived'].mean():.1%}")

print("\n⚠️  MISSING DATA:")
print(f"  - Age: {df['Age'].isnull().sum()} missing (~20%) → Imputed with median")
print(f"  - Cabin: 77% missing → Converted to Has_Cabin binary flag")

print("\n🔺 TRENDS IDENTIFIED:")
print("  - Survival decreases with passenger class (1st > 2nd > 3rd)")
print("  - Younger children (0–12) show highest survival rate")
print("  - Higher fare passengers consistently more likely to survive")

print("\n🔷 PATTERNS IDENTIFIED:")
print("  - Women consistently outsurvived men across ALL classes")
print("  - Medium-sized families (2–4) survived better than solo or large groups")
print("  - Title 'Mrs' and 'Miss' strongly associated with survival")

print("\n🚨 OUTLIERS:")
print("  - 116 fare outliers detected via IQR (max: $512.33)")
print("  - Recommend log-transformation of Fare before modeling")

print("\n🆕 ENGINEERED FEATURES:")
print("  - Title_Group (extracted from Name)")
print("  - Family_Size = SibSp + Parch + 1")
print("  - Is_Alone (binary flag)")
print("  - Has_Cabin (binary flag)")
print("  - Age_was_missing (binary flag)")

The Expert EDA Mindset — A Checklist

Before calling your EDA complete, ask yourself:

• [ ] Do I know the shape, types, and basic stats of every column?
• [ ] Have I mapped and handled all missing values with intent (not just filled blindly)?
• [ ] Have I visualized distributions for all key numerical features?
• [ ] Have I checked for skewness and considered transformations?
• [ ] Have I identified both trends and patterns in the data?
• [ ] Have I created a correlation matrix and understood it?
• [ ] Have I detected and made a deliberate decision about outliers?
• [ ] Have I extracted at least 2–3 new features from existing ones?
• [ ] Can I tell the data’s story in plain English to someone non-technical?

“The goal of data analysis is to extract signal from noise. EDA is how you first learn to hear the signal.” — Cassie Kozyrkov, Chief Decision Intelligence Engineer, Google

Conclusion — The Skill That Multiplies Every Other Skill

Here’s the thing nobody tells you when you start out: your model is only as good as your understanding of the data it’s built on.

A decision tree trained on raw Titanic data will give you something. A decision tree trained after rigorous EDA — with clean missing values, log-transformed Fare, a hand-crafted Title_Group feature, and a Family_Size variable — will give you something significantly better.

But more importantly, EDA teaches you humility. It forces you to sit with the data before you make assumptions. It trains you to ask “why?” before you ask “how?” It’s the difference between a scientist and someone running code.

The gap between a beginner and an expert in data science isn’t about knowing more algorithms. It’s about spending more quality time with the data before ever touching one.

So the next time you download a fresh dataset, resist the urge to jump straight to the model. Open it up. Look at it. Ask it questions. Let it surprise you.

The patterns are already there. EDA is just how you learn to listen.

📚 Further Reading

• “Exploratory Data Analysis” — John W. Tukey (1977) — the book that started it all
• “Python for Data Analysis” — Wes McKinney (creator of pandas)
• “Storytelling with Data” — Cole Nussbaumer Knaflic
• Kaggle’s Titanic notebook community — hundreds of EDA examples in practice

Authored by: Shorya Bisht