A Complete Guide to Python Exception Handling for Data Science

The Unsung Hero of Python for Data Science

Image generated using DALL-E

Table of Contents

1. Why exception handling?
2. Understanding when errors happen
3. Understanding why errors happen
4. Frequently encountered exceptions in data science workflows
5. Practical ways to safely handle exceptions in Python
6. Conclusion

Why Exception Handling?

A couple of years ago, when I began my data science journey, learning Python programming was one of my top priorities, as a significant portion of my work required Python. Gradually, I developed proficiency in many of the Python concepts, but I overlooked a few key ones. The simplest reason was that I didn’t find its use case directly in data science operations, such as data loading, preprocessing, analysis, and model building. I could perform insightful and in-depth analysis, build good models, but once faced a big problem — a tiny error, and the code failed. One small, unexpected input broke everything. That’s where I understood the importance of exception handling, one such concept I had previously overlooked.

Today, when I interact with fresh minds entering the world of data science, machine learning, and artificial intelligence, I hear them discussing model architectures, hyperparameters, accuracy, and so on, but I rarely hear them discuss exception handling. So, I decided to share what I know about exception handling and how I use this silent superpower to save hours of debugging.

Now, since I have created enough curiosity about why understanding and handling exceptions is important, let’s start diving deep into the topic.

Understanding When Errors Happen

There are two stages where errors can occur:

1. Compilation (Syntax errors)
2. Execution (Runtime errors)

Syntax errors occur before the program is run. It occurs due to mistakes made in the code's structure, such as missing parentheses, indentation errors, mispelled keywords, or incorrect library names.

Runtime errors occur during the execution of the program. The code written is syntactically correct, but the program fails due to some unexpected situations. For example, accessing a missing data file, dividing a value by zero, etc. Runtime errors are the most common type of error we as data scientists usually encounter, and exception handling proves to be the best defense mechanism against these errors.

Understanding Why Errors Happen

Whenever we build a data science project, we expect clean data files without any inconsistencies. Well, it's a dream come true if this happens. But the most unfortunate part of the process is that the data is rarely clean. It contains all kinds of messiness — incomplete entries, inconsistent formatting, inaccurate values, and sometimes, the data is not properly structured. In short, we can say that the nature of the dataset we receive is unpredictable. So, let’s understand quickly, why does this happen?

There are a few reasons why data is messy:

1. Man-made errors like leaving certain fields blank, misspelled entries, and entering data in the wrong format, etc.
2. Data coming from different systems can lead to inconsistencies.
3. Data may contain impossible values.
4. Unexpected input entered into data pipelines.
5. Data drift: data changing over time, etc.

Frequently Encountered Exceptions in Data Science Workflows

As we discussed why exceptions occur, the reasons for the same can be unpredictable. So, as a data scientist, it is important to build robust workflows and try to minimize the risks of the programs crashing due to unexpected exceptions. So let’s understand some of the most important exceptions encountered while building data science projects. For gaining practical understanding, I have used a sample dataset, which you can download from here.

ParserError

ParserError is one of the most common errors encountered in data science workflows. It is triggered when the text does not follow the expected format.

Some of the most common causes include:

• Inconsistent number of columns in a row
• Wrong delimiter assumptions
• Mixed line endings, or broken lines, etc.

Below is one such practical example for better understanding:

data_path = "/content/drive/MyDrive/Medium/hr_dataset.csv"
df = pd.read_csv(data_path)
df.info()

Error message:

ParserError: Error tokenizing data. C error: Expected 6 fields in line 13, 
saw 7

This error occurred due to an inconsistent number of columns found in line 13. It expected 6 but found 7. These kinds of errors are very common, and as a responsible data scientist, it is important to create robust measures to handle such errors.

FileNotFound Error

FileNotFoundError occurs when Python tries to access a file that is not present at the specified location. This is again pretty common when we work on real-world projects.

The most common causes include:

1. Sometimes, the file names are misspelled.
2. Incorrect location provided.
3. We run code from a different working directory.
4. The file has been moved to a different location, deleted, or renamed — (most common).
5. There are some cloud or network storage issues, etc.

Let’s look at a realistic scenario:

data_path = "/content/drive/MyDrive/British Airways/hr_merged_clean.xlsx"
df = pd.read_excel(data_path)
df.head()

Error Message:

FileNotFoundError: [Errno 2] No such file or directory:
 '/content/drive/MyDrive/British Airways/hr_merged_clean.xlsx'

The error occurred because Python could not find the data file at the specified data path. The file may have been moved to a different location.

ImportError or ModuleNotFoundError

When working on a data science project, ImportError or ModuleNotFoundError (subclass of ImportError) are very common because data science workflows involve many dependencies, environments, libraries, and external tools.

Most common causes include:

1. The library is either not installed or not installed correctly in the environment.
2. The module name is misspelled.
3. Production environments do not match the development environments.
4. Dependencies are missing in the installed libraries.
5. Multiple Python files import each other, etc.

Let’s look at a practical occurrence of this error:

import featuretools

Error message:

ModuleNotFoundError: No module named 'featuretools'

Here, as we can observe, the script uses featuretools library, but since featuretools is not installed in our environment, Python raises the ModuleNotFoundError. A data scientist must handle such situations carefully to prevent the system from failing unexpectedly.

KeyError

Python raises a KeyError When we try to access a column or an index that does not exist. It simply means that when Python tried to access the specified column or index, it could not find it and eventually raised the error.

It is one of the most common and frequently encountered errors that is caused by a variety of reasons:

1. When we try to access a non-existent Pandas column.
2. When we try to access a column that was earlier dropped during preprocessing.
3. Sometimes, merging datasets from multiple sources introduces missing keys.
4. When categories are renamed or deleted, etc.

Let’s see how this error appears in practice:

data_path = "/content/drive/MyDrive/Medium/hr_dataset.csv"
df = pd.read_csv(data_path, on_bad_lines="skip")

df = df.rename(columns={"employee_id": "id"})
df["employee_id"]

Error message:

KeyError: 'employee_id'

In the above example, we can see that we tried to access the employee_id column, which was previously renamed to id raising the KeyError. Such instances are very common in real-world applications, where a column is renamed during preprocessing, but is accidentally referenced by its original name.

IndexError

IndexError occurs when we try to access a position in a list, array, or DataFrame that doesn’t exist. IndexError is frequently encountered during data preparation, feature engineering, and ML preprocessing.

The most common causes IndexError include:

1. Accessing a row index that does not exist.
2. Accidentally dropping rows during cleaning.
3. Flattening or reshaping arrays incorrectly.
4. Referencing columns or rows that were removed earlier.
5. Inconsistent indexes after concatenating DataFrames, etc.

Let’s understand with a scenario:

Suppose the HR department decided to award a randomly numbered employee in each salary bracket with a bonus. For the salary bracket of 70000 and above, the tenth employee has to be rewarded.

data_path = "/content/drive/MyDrive/Medium/hr_dataset.csv"
df = pd.read_csv(data_path, on_bad_lines="skip")

temp = df[df["salary"] > 70000]
temp.iloc[10]

Error message:

IndexError: single positional indexer is out-of-bounds

In the above example, we can see that we filtered the rows based on the condition that the salary should be greater than 70000, but since there were fewer than 11 employees who had a salary of 70000 or above, Python raised an IndexError.

TypeError

TypeError occurs when we try to perform an operation on a variable whose data type is not compatible with the operation. Since real-world datasets are messy and Python has strict rules for each of its data types, these errors are very common in data science workflows.

The most common causes include:

1. Features containing mixed data types (strings, numbers, special characters, etc.).
2. Applying operations not compatible with the data type. For example, performing arithmetic operations on non-numeric features.
3. Feeding incorrect data types as arguments to ML models.
4. Accidental type conversions in DataFrame columns.
5. Performing operations on missing values, etc.

Let’s understand the error with a simple example:

Suppose the HR department decided to give a salary increment of Rs. 5000 to every employee, rewarding their exceptional performance.

data_path = "/content/drive/MyDrive/Medium/hr_dataset.csv"
df = pd.read_csv(data_path, on_bad_lines="skip")

df["salary"] = df["salary"] + 5000

Error message:

TypeError: can only concatenate str (not "int") to str

We encountered the above error because some values in the salary column are stored as strings, and Python cannot perform arithmetic operations on string values, and thus raised the TypeError.

ValueError

ValueError occurs when invalid values, though of a correct data type, are provided to functions or models. Messy real-world datasets often contain mixed data types, unexpected values, etc.

The most common causes include:

1. Numeric columns might contain strings that cannot be converted.
2. Invalid date formats while parsing dates.
3. Providing invalid inputs to ML models. For example, some metrics such as Root Mean Squared Logarithmic Error (RMSLE), Poisson Deviance, Log Loss, Mean Absolute Percentage Error (MAPE), etc.
4. Feeding wrong shapes to ML algorithms, etc.

Here’s how this looks in code:

data_path = "/content/drive/MyDrive/Medium/hr_dataset.csv"
df = pd.read_csv(data_path, on_bad_lines="skip")

df["salary"] = df['salary'].astype(int)

Error message:

ValueError: invalid literal for int() with base 10: 'Seventy Five Thousand'

The above error occurred because Python tried to convert a string value (Seventy Five Thousand) to an integer. In real-world scenarios, users might provide invalid input, and if such values are not handled properly, the program can fail unexpectedly.

AttributeError

As a data scientist, we work with a variety of object types in Python, such as Numpy arrays, Pandas series and DataFrames, Python lists, dictionaries, models, transformers, etc. When we try to access an attribute or method that does not belong to an object, Python raises the AttributeError.

The most common scenarios where AttributeErroroccurs in data science workflows:

1. Pandas series and DataFrame objects have many similar attributes. Thus, it is quite common to use a series-specific attribute on a DataFrame and vice versa.
2. Attribute or method names are spelled incorrectly.
3. Chaining operations incorrectly.
4. Using model methods that don’t exist.
5. In the case of merging, selecting, or grouping objects, the structure of the resultant object changes, etc.

Let’s understand this better with an example:

data_path = "/content/drive/MyDrive/Medium/hr_dataset.csv"
df = pd.read_csv(data_path, on_bad_lines="skip")

df["name"].upper()

Error message:

AttributeError: 'Series' object has no attribute 'upper'

In the above example, we want to check the data types of each feature in our DataFrame. Unknowingly, we used the “dtype” attribute, which is a series attribute, instead of “dtypes,” which raised the AttributeError. Such tiny errors are often encountered while working on data science projects.

NameError

NameErroroccurs when Python sees a variable, function, or model name that is not defined in the current scope. Since we work on Jupyter notebooks with many cells, the probability of encountering NameErroris quite high.

NameError can be encountered in a variety of scenarios. Some of them are mentioned below:

1. When we use a variable before defining it.
2. When there is a typo in variable or function names.
3. When we use a function or a module without importing it.
4. When variables or functions are overwritten or deleted in the workflow, etc.

The following code snippet clearly demonstrates the issue:

data_path = "/content/drive/MyDrive/Medium/hr_dataset.csv"
df = pd.read_csv(data_path, on_bad_lines="skip")

temp_df = df[df["salary"] > 70000]
tempdf.head()

Error message:

NameError: name 'tempdf' is not defined

We encountered a NameError here because we accidentally used “tempdf” instead of “temp_df”.

Practical Ways to Safely Handle Exceptions in Python

So far, we have discussed that real-world datasets are messy and unpredictable, and how this inherent uncertainty makes data science workflows vulnerable to errors and failures. Think of these workflows as scientific experiments. When something unexpected happens during these experiments, scientists don’t stop them. Rather, they record the errors, identify their probable causes, and adjust the experiments accordingly. Similarly, as data scientists, we run the pipelines, encounter errors, and implement exception handling techniques to make our pipelines robust to errors.

Now, I will share some key exception handling techniques that I frequently use while building data science workflows:

1. Using Inline Fixes

If you noticed in the previous examples where I have given code examples while explaining the errors, I have used on_bad_lines="skip" parameter when reading the csv file. You can see the complete code below:

data_path = "/content/drive/MyDrive/Medium/hr_dataset.csv"
df = pd.read_csv(data_path, on_bad_lines="skip")

Many functions in Pandas and other libraries have parameters that handle bad entries inside the functions themselves. One more example can be:

df["Date"] = pd.to_datetime(df["Date"], errors="coerce")

Here, too, erroneous date values are handled internally by the function, and error values are converted to NaNs (Not a Number).

These error-handling parameters can be useful in certain scenarios, but are not very robust. These techniques can lead to loss of information, data corruption, statistical bias, improper validation, and the absence of logging. Therefore, I don’t advise using such techniques.

2. Try-Except-Else-Finally

Adding try-except blocks is one of the most commonly used techniques to handle errors. It prevents the workflow from failing abruptly due to errors.

Let’s understand the structure and use of the try-except-else-finally block with an example:

try:

  # code that may raise an exception
  df['salary'] = pd.to_numeric(df['salary'], errors='raise')

except ValueError as e:

  # handle ValueError
  print("Cannot convert string to integer")

except Exception as e:

  # handle general error
  print(e)

else:

  # runs only if no exception is raised
  df['salary'] = df['salary'].astype(int)

finally:

  # The code you want to run irrespective of errors occuring or not
  print("Preview of the data")
  print(df.head())

The try block is generally used for handling file operations, network calls, type conversions, database queries, etc., or anything involving external or uncertain data. Here, we are using it to convert the salary column to an integer.

The except block catches the errors thrown by the try block. It can handle single or even multiple exceptions. It is generally used for logging and understanding errors, and providing a safe alternative action when the program fails. As you can see in the above example, we are using it to catch the ValueError and in case there are some other unexpected errors.

The else block contains the code that is executed when no error is encountered. It is used to separate the error-prone code from safe code. In our example, it contains the code to convert the salary column to an integer, if there are no errors.

The finally block contains the code that has to be executed irrespective of any errors occurring or not. It is generally used for tasks such as closing file handles and database connections, shutting down API sessions, etc. In our example, we are previewing our data irrespective of errors occurring.

The key advantage of using try-except blocks is that it allows targeted error handling, i.e., it allows catching only those errors that are important for the program.

3. Raising Exceptions

Raising exceptions means intentionally stopping the program flow when something unexpected is detected that should not be allowed to continue. It’s like a fire alarm — when it detects danger, it interrupts everything.

In data science workflows, exceptions are raised for a variety of reasons. For example, if there are some unexpected errors in data, or API failures are detected, or the user inputs invalid data in ML applications, or some structural changes occur in the schema, exceptions are raised to stop the workflow at the point where the problem is detected.

Let’s understand how to raise exceptions with a realistic scenario:

Suppose we have written a function to convert the age column to an integer. But the problem is that the column contains missing values. Using the “raise” keyword, we can raise the ValueError in the following way:

def convert_age_to_int(df, age_column):
  numeric_age = pd.to_numeric(df[age_column], errors='coerce')

  if numeric_age.isna().any():
    raise ValueError("Cannot convert non-numeric data to integer")

  else:
    df[age_column] = numeric_age.astype(int)
    return df

convert_age_to_int(df=df, age_column="age")

Error message:

ValueError: Cannot convert non-numeric data to integer

The major advantage of raising exceptions is that it catches data issues early, preventing model failures.

4. Custom Exceptions

Custom exceptions are used to describe errors specific to the domain we are working on.

Let’s demonstrate it with an example:

Suppose we want to check whether the salary column in our dataset contains some invalid values or not:

The very first step is to create the custom exception. To create custom exceptions, it is necessary to inherit from the base Exception class.

class InvalidSalaryFormatError(Exception):
    """Raised when salary column contains invalid or non-numeric values."""
    pass

The InvalidSalaryFormatError exception is raised when it detects non-numeric or invalid values in the salary column.

def validate_salary_column(df):
    # checks the salary column and detects invalid values
    invalid_entries = df[~df["salary"].astype(str).str.replace(".", "", 1).str.isdigit()]

    if not invalid_entries.empty:
        raise InvalidSalaryFormatError(
            f"Invalid salary values detected:\n{invalid_entries[['employee_id','salary']]}"
        )

What the function does is that it converts the salary column to a string, checks whether each value is numeric, finds invalid entries, and raises the InvalidSalaryFormatError with the context, if an error is detected.

try:
    validate_salary_column(df)
    df["salary"] = df["salary"].astype(float)
    print("Salary column validated and converted successfully.")

except InvalidSalaryFormatError as e:
    print("Custom Error: Salary validation failed.")
    print(e)

except Exception as e:
    print("An unexpected error occurred:", e)

finally:
    print("\nData Preview:")
    print(df.head())

Finally, we used the try-except-else-finally block to catch the exception. It showed the following result when executed:

Custom Error: Salary validation failed.
Invalid salary values detected:
   employee_id                 salary
2            3  Seventy Five Thousand
3            4                    NaN

Data Preview:
   employee_id   name   age                 salary   department       city
0            1   Asha  29.0                65000.0  Engineering  Bengaluru
1            2   Ravi  35.0                85000.0        Sales     Mumbai
2            3  Meera  28.0  Seventy Five Thousand           HR      Delhi
3            4  Arjun   NaN                    NaN      Finance    Chennai
4            5  Divya  31.0                72000.0          NaN        NaN

5. Logging

In my experience as a data scientist, I have found logging to be one of the most powerful exception-handling techniques. Logging does not prevent errors, but enables safe recovery by diagnosing and recording errors for later inspection.

Let’s understand logging with a realistic scenario:

import logging
# setting the logger
logger = logging.getLogger("hr_data")

# Remove previously added handlers
logger.handlers.clear()
logger.setLevel(logging.DEBUG)
logger.propagate = False

# setting the handler
handler = logging.StreamHandler()
handler.setLevel(logging.DEBUG)

# setting the formatter
formatter = logging.Formatter(
    "%(asctime)s - %(name)s - %(levelname)s - %(message)s - EXTRAS: %(extra)s"
)

handler.setFormatter(formatter)
logger.addHandler(handler)

def validate_salary(df):
  coerced = pd.to_numeric(df['salary'], errors='coerce')
  invalid = df[coerced.isna() & df['salary'].notna()]

  if not invalid.empty:
    logger.error(
        "Non-numeric data found in salary column",
        extra={"extra": {"invalid_count": int(len(invalid))}}
    )

    for idx, val in invalid['salary'].items():
      logger.warning("Invalid salary",
                     extra={"extra": {"row": int(idx), "val": str(val)}})

    raise InvalidSalaryFormatError(f"{len(invalid)} invalid records found")

  else:
    df['salary'] = coerced.astype(float)
    logger.info("Salary conversion successful")
    return df  

try:
  validate_salary(df)
except InvalidSalaryFormatError as e:
  logger.critical("Salary conversion failed", extra={"extra": {}})

I have used the previously demonstrated validate_salary example, using logging instead. In the first part, I have set up the logger, handler, and the formatter. Then, used logging to trace where errors happen, and subsequently printed them to the console. Executing the code yields this output:

2025-12-12 10:47:54,806 - hr_data - ERROR - Non-numeric data found in salary column - EXTRAS: {'invalid_count': 1}
2025-12-12 10:47:54,807 - hr_data - WARNING - Invalid salary - EXTRAS: {'row': 2, 'val': 'Seventy Five Thousand'}
2025-12-12 10:47:54,808 - hr_data - CRITICAL - Salary conversion failed - EXTRAS: {}

You will observe how well the errors have been diagnosed and recorded. This makes it easier to understand where things went wrong. This is why logging is preferred by most data scientists when building production-ready workflows.

Conclusion

In my opinion, exception handling is a foundational skill that helps data scientists to build resilient, trustworthy, and production-ready workflows. As we glanced through the common errors encountered in data science workflows, we understood that errors can surface at any stage of the workflow, and by adopting structured exception handling techniques, we can not only detect issues early but also recover gracefully without disrupting downstream tasks. Thus, mastering exception handling enables data scientists to build robust, scalable, and reliable data science solutions.

Thanks for investing your time to read the article!

— Vandan Rana