Handling Missing Data and NaN Values in NumPy

Hey guys! Welcome back to our **NumPy for DS & DA series. This is going to be the 6th article of this series. In our previous article, we discussed [Mathematical and Statistical Functions](https://nsdsda.medium.com/mathematical-and-statistical-functions-in-numpy-d846c0f2fbec) **in NumPy. We also saw some code examples to understand the concepts better. So, if you haven’t yet read the previous articles, you can check them out first.

So, let’s get into our today’s topic now.

In DS & DA, real-world data is almost never perfect. Missing values, which we also call as NaNs (Not a Number), are very common. Handling these Missing Values the right way is very important before performing any Analysis or building Models.

Handling Missing Data and NaN Values in NumPy

In this article, let’s explore how to Detect, Handle, and Clean missing data using NumPy. I’ll also give you some practical examples so that you get the concepts better.

Before we start, we know that we’re going to work with NumPy, no matter what topic we’re working with. So, let’s first import NumPy and get into it.

import numpy as np

What is NaN in NumPy?

NaN is basically Not a Number and is used to represent missing or undefined values. We use “np.nan” to represent these values in NumPy.

arr = np.array([1, 2, np.nan, 4, 5])
print(arr) # Output: [ 1.  2. nan  4.  5.]

Detecting NaN Values

In NumPy, we have a built-in function “np.isnan()” to check for NaN values in an array. This will return a Boolean array, means False if the value is a number, and True if not.

arr = np.array([1, 2, np.nan, 4, 5])

mask = np.isnan(arr)
print(mask) # Output: [False False  True False False]

Counting Missing Values

To find how many missing values are in the array, we can use “np.sum()” which counts how many True values the mask contains, or how many NaNs are present.

print(np.sum(np.isnan(arr))) # Output: 1

We got 1 as output, as we have only 1 NaN in our array.

Removing Missing Values

This is where, we’re going to use Boolean Masking.

clean_arr = arr[~np.isnan(arr)]
print(clean_arr) # Output: [1. 2. 4. 5.]

The symbol ~ means “not”. So, this selects only the elements where “isnan()” is False, means the valid values.

Replacing Missing Values

Sometimes, instead of dropping missing values, we need to fill them with something meaningful, like Mean, Median, or a fixed value.

Let’s replace our NaN with Mean here. For this, we’ll use “nanmean()” method, which basically calculates the Mean of the values, ignoring NaN.

mean_val = np.nanmean(arr)
arr[np.isnan(arr)] = mean_val

print(arr) # Output: [1. 2. 3. 4. 5.]

Here, we replaced the missing value with 3.0, which is the Mean/Average of the valid values.

Mathematical Functions & NaNs

Standard NumPy functions like “np.sum()”, “np.mean()”, “np.std()”, or “np.var()” return NaN if even one NaN exists, right?

So, to ignore NaNs in such cases, we can use NaN-safe functions. I know NaN-safe sounds weird.😂

Anyway! Here are a few common functions, we can use instead, to avoid such situations when dealing with NaNs.

| Regular Functions | NaN-safe Alternatives |
|-------------------|-----------------------|
| np.sum()          | np.nansum()           |
| np.mean()         | np.nanmean()          |
| np.std()          | np.nanstd()           |
| np.var()          | np.nanvar()           |
| np.min()          | np.nanmin()           |
| np.max()          | np.nanmax()           |

arr = np.array([10, np.nan, 15, 16, np.nan, 24])

print(np.nanmean(arr))   # Output: 16.25
print(np.nansum(arr))    # Output: 65.0

These functions ignore missing values automatically.

Missing Data in Multi-Dimensional Arrays

We can use all of these techniques on 2D arrays as well, they’ll work the same on them. We’ll get a Boolean mask showing where each NaN is.

arr2d = np.array([[1, 2, np.nan], [4, np.nan, 6]])

print(np.isnan(arr2d))
# Output: 
[[False False  True]
 [False  True False]]

We can also apply “np.nanmean()” along an axis.

print(np.nanmean(arr2d, axis=0))  # Output: [2.5 2.  6. ]
print(np.nanmean(arr2d, axis=1))  # Output: [1.5 5. ]

Here “axis=0” goes column-wise, that means, the function is applied across all rows for each column. And “axis=1” goes row-wise (across columns for each row).

Why Handling Missing Data Matters?

• Missing values can break Statistical Calculations.
• They can distort ML Models if not handled carefully.
• A proper approach depends on Business Context and Data Size.

In real projects, we often combine,

• Removing NaNs if the missing data is tiny.
• Replacing NaNs using Mean/Median/Mode or other Techniques.

Missing data is inevitable in real-world datasets. I know I’m using some fancy words.😂 But okay!

With NumPy’s methods like “isnan()”, “nanmean()”, and Boolean masking, we can handle them efficiently and confidently.

As a Data professional, these are essential skills that you’ll use all the time. So the next time you see a NaN, you know exactly what to do!😁

And that’s all for today!

Check out related Lists:

• **NumPy for DS & DA**
• **Pandas for DS & DA**
• **Intro to Statistics**

Thanks for reading! 😊

A message from our Founder

Hey, Sunil here. I wanted to take a moment to thank you for reading until the end and for being a part of this community.

Did you know that our team run these publications as a volunteer effort to over 3.5m monthly readers? We don’t receive any funding, we do this to support the community. ❤️

If you want to show some love, please take a moment to follow me on LinkedIn, TikTok, **Instagram. You can also subscribe to our [weekly newsletter](https://newsletter.plainenglish.io/)**.

And before you go, don’t forget to clap and follow the writer️!