Pandas: First Steps with Pandas

Welcome to the second post in my Python Library Mastery series! Pandas!

Today we’re diving deep into the two most important (and honestly, most satisfying) parts of any data workflow: Data Cleaning and Data Transformation. And it will be in two different blogs. First in this we will be preparing and seeing what we have. Next, we will be cleaning and transforming.

If you work with data, you already know the pain and the joy of turning messy, chaotic spreadsheets into something beautiful and usable.

And if you love Python, you know there’s no better tool for the job than pandas.

But for the beginners in the room:

What exactly are data cleaning and transformation?

In simple words:

Data Cleaning = Removing the dirt (missing values, duplicates, wrong formats, outliers, typos, garbage).

Data Transformation = Reshaping the data into the exact structure you need for analysis (changing data types, creating new columns, normalizing, merging, pivoting, etc.).

Or, as I like to explain it with a real-life analogy:

Imagine your kitchen cupboard is full of brand-new utensils: plates, bowls, spoons, forks… everything is mixed up, some still have price tags, a few are cracked, and half the spoons are bent.

Before cooking an amazing dinner (your analysis/model), you:

Wash everything = Data Cleaning

Sort them properly, maybe turn a deep plate into a soup bowl = Data Transformation

That’s it. Clean it. Shape it. Then cook insights.

Formal definitions (for the resume crowd):

“Data cleaning is the process of fixing or removing incorrect, corrupted, incorrectly formatted, duplicate, or incomplete data within a dataset.” — IBM

“Data transformation converts data from one format or structure into another to make it more appropriate for analysis.” — IBM

And when I asked Perplexity AI, it gave the perfect summary:

“Together, cleaning and transformation turn raw, messy data into accurate, consistent, and analysis-ready gold.”

Now, here’s the fun part:

We’re going to do all of this like a panda.

“The first step to mastering chi is to know who you are. Well, I’m a panda.”

— Po, Kung Fu Panda

So get ready to code like a panda, think like a panda, roll like a panda and eat messy datasets for breakfast.

The Dataset We’ll Be Using Throughout This Series: The Dinosaur Dataset from Kaggle!

I’ve wanted to play with this ridiculously fun dataset forever. Finally, here’s my excuse. Expect columns like name, length, diet, period, country found, and lots of missing or weird values, perfect for real-world cleaning practice.

Am I excited?

100%

So fire up your Jupyter notebook, Google Collab, Microsoft VScode whatever you like, do a quick,

import pandas as pd

And let’s turn into data kung-fu pandas together.

Let’s begin!

First we will meet our dataset. What is in this? What are the columns? What types of data we have?

At first glance and studying the description of the Kaggle description, what we are seeing is:

This dataset is full of fossil insights. We have the columns below.

Now we will see it through pandas. First we will create a Dataframe:

df = pd.read_csv("dinosaurs.csv")
df.head(10)

This is what our Dataset looks like. df.head(10) shows us the first 10 rows of dataset. By default, it shows top 5.

The same way we can see the last 5 data using df.tail(). The same way df.tail() by default shows the last 5. But if we put any number in between the brackets, that’s the number of how many rows we want to see.

We already know what the columns are by reading the Kaggle description but if we want to see through pandas dataframe then we can see it using df.columns. And if we want to see it as a list, then just add .tolist() at the end.

df.dtypes shows the data types of the columns.

Now let’s run a quick statistical summary using df.describe(). This will only use the numeric data type columns like occurrence_no, length_m, max_ma, min_ma, lng and lat and give a descriptive summary of the columns.

Let’s read what each row represents:

count: Number of non-null values. Notice length_m has only 3,568 but max_ma, min_ma, lng and last has 4,951. So around, 1,383 missing values.

mean: Average value across all row.

sstd: Standard deviation, how spread out values are from the mean. Large std = high variability

min: Smallest value in the column

25%: 25th percentile (Q1), 25% of values fall below this

50%: Median, the middle value (more robust than mean for skewed data)

75%: 75th percentile (Q3), 75% of values fall below this

max: Largest value in the column

What each column gives use about the dataset?

occurrence_no: This is likely an ID column (scientific notation: Around 130K to Around 1.37M). Mean/std aren't meaningful here.

length_m: Notice length_m has only 3,568 but max_ma, min_ma, lng and last has 4,951. So around, 1,383 missing values. Range is 0.45m -> 35m, median is 6.7m. Slightly right-skewed (mean 8.2 > median 6.7) Meaning pulled up by high outliers.

max_ma / min_ma: Geological age in millions of years. Range around 66–252 Ma (Cretaceous to Permian). The 25th percentile at around 83.5 Ma suggests most occurrences are Cretaceous.

lng / lat: Geographic coordinates. Notice lng has a max of 565, that’s impossible (valid range: -180 to +180), signaling a data quality issue worth investigating.

There is a Quick Rule of Thumb to read Summary Statistics:

• mean ≈ median -> data is roughly symmetric
• mean > median -> right-skewed (pulled up by high outliers)
• mean < median -> left-skewed (pulled down by low outliers)
• std > mean -> very high variability, possibly outliers
• count < total rows -> missing values exist in that column

The statistical test can be done in another

df.describe(include="all")

So this is an upgrade from the first df.describe(). This has an extra “all” in it. So what this does is, this gives me the descripted summary of every column. Not just the numeric ones.

Numeric columns (occurrence_no, length_m, max_ma, min_ma, lng, lat) still giving mean, std, min, percentiles, max, same as before. Their unique, top, and freq rows are NaN because those concepts don’t apply to continuous numbers (every value could be “unique,” so it’s not informative).

But the word like “herbivorous” can’t be averaged. So for text columns (name, diet, type, region, class, family), pandas switches to a different set of questions:

1. unique = how many different answers showed up? (diet only has 3 possible answers, so it’s basically a small set of categories. name has 1042 different dinosaur names, so clearly some dinosaurs show up more than once in the data.)
2. top = which answer showed up the most? (For diet, that’s “herbivorous.”)
3. freq = how many times did that top answer show up? (herbivorous appeared 2076 times, so plant-eaters make up a big chunk of this dataset.)

Everywhere a number-stat doesn’t make sense for text (like “mean” of a word), pandas just leaves it blank (NaN), and everywhere a text-stat doesn’t make sense for numbers (like “top” of a column of decimals), it’s blank too.

The one-line way to remember it: plain describe() summarizes your numbers, describe(include=”all”) summarizes everything, just in two different “languages” stitched into one table, with blanks wherever a language doesn’t apply.

The next and last check we will do is

df.info()

This one’s a nice complement to describe(), it’s less about statistics and more about structure: how many rows, how many columns, what type each column is, and how many values are actually filled in.

What the output is saying?

1. This dataframe got 4951 entries (rows), indexed 0 to 4950, and 12 columns total.
2. Then each row in the table says that per column, how many non-null values it has and what data type pandas assigned it.
3. So the outputs are:

name: 4951 non-null, fully complete diet: 3596 non-null, so 1355 missing values type: 3596 non-null, same gap, 1355 missing length_m: 3568 non-null, matching the 1383-missing figure from your describe() walkthrough region: 4909 non-null, so 42 rows have no region family: 3494 non-null, so 1457 missing

Everything else (occurrence_no, max_ma, min_ma, lng, lat, class) is fully populated at 4951.

The dtype column is the other half of the story. int64 and float64 are the number columns (whole numbers and decimals), and object is pandas’ catch-all for text/strings.

This is a handy sanity check too, sometimes a column that should be numeric gets read in as object because of stray text or formatting issues in the raw CSV, so it’s worth a quick eyeball here before moving on to cleaning.

And that memory usage line at the bottom just says the DataFrame is taking up about 464 KB in memory, not something need to be worried about with a dataset this size, but it becomes relevant once much larger files is being handled.

So while describe() says the shape of numbers of the dataframe, info() says the shape of the dataset, and together they’re basically “first date” checklist before cleaning anything.

So this has been our dataset. Our first look. When we understand how the dataset looks like, we are going to take decisions to how to clean and transform the dataset before doing any cleaning. So the next article will be focusing on what we will be making decision about the process of cleaning and transformation!